Structuring, aggregating, and evaluating crowdsourced design critique

Kurt Luther; Jari Lee Tolentino; Wei Wu; Amy Pavel; Brian P. Bailey; Maneesh Agrawala; Björn Hartmann; Steven P. Dow

doi:10.1145/2675133.2675283

Structuring, aggregating, and evaluating crowdsourced design critique

Kurt Luther, Jari Lee Tolentino, Wei Wu, Amy Pavel, Brian P. Bailey, Maneesh Agrawala, Björn Hartmann, Steven P. Dow

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Feedback is an important component of the design process, but gaining access to high-quality critique outside a classroom or firm is challenging. We present CrowdCrit, a webbased system that allows designers to receive design critiques from non-expert crowd workers. We evaluated CrowdCrit in three studies focusing on the designer's experience and benefits of the critiques. In the first study, we compared crowd and expert critiques and found evidence that aggregated crowd critique approaches expert critique. In a second study, we found that designers who got crowd feedback perceived that it improved their design process. The third study showed that designers were enthusiastic about crowd critiques and used them to change their designs. We conclude with implications for the design of crowd feedback services.

Original language	English (US)
Title of host publication	CSCW 2015 - Proceedings of the 2015 ACM International Conference on Computer-Supported Cooperative Work and Social Computing
Publisher	Association for Computing Machinery, Inc
Pages	473-485
Number of pages	13
ISBN (Electronic)	9781450329224
DOIs	https://doi.org/10.1145/2675133.2675283
State	Published - Feb 28 2015
Event	18th ACM International Conference on Computer-Supported Cooperative Work and Social Computing, CSCW 2015 - BC, Canada Duration: Mar 14 2015 → Mar 18 2015

Publication series

Name	CSCW 2015 - Proceedings of the 2015 ACM International Conference on Computer-Supported Cooperative Work and Social Computing

Other

Other	18th ACM International Conference on Computer-Supported Cooperative Work and Social Computing, CSCW 2015
Country	Canada
City	BC
Period	3/14/15 → 3/18/15

Keywords

Design; critique; feedback; social computing; crowdsourcing

ASJC Scopus subject areas

Computer Networks and Communications

Access to Document

10.1145/2675133.2675283

Fingerprint Dive into the research topics of 'Structuring, aggregating, and evaluating crowdsourced design critique'. Together they form a unique fingerprint.

Cite this

Luther, K., Tolentino, J. L., Wu, W., Pavel, A., Bailey, B. P., Agrawala, M., Hartmann, B., & Dow, S. P. (2015). Structuring, aggregating, and evaluating crowdsourced design critique. In CSCW 2015 - Proceedings of the 2015 ACM International Conference on Computer-Supported Cooperative Work and Social Computing (pp. 473-485). (CSCW 2015 - Proceedings of the 2015 ACM International Conference on Computer-Supported Cooperative Work and Social Computing). Association for Computing Machinery, Inc. https://doi.org/10.1145/2675133.2675283

Structuring, aggregating, and evaluating crowdsourced design critique. / Luther, Kurt; Tolentino, Jari Lee; Wu, Wei; Pavel, Amy; Bailey, Brian P.; Agrawala, Maneesh; Hartmann, Björn; Dow, Steven P.

CSCW 2015 - Proceedings of the 2015 ACM International Conference on Computer-Supported Cooperative Work and Social Computing. Association for Computing Machinery, Inc, 2015. p. 473-485 (CSCW 2015 - Proceedings of the 2015 ACM International Conference on Computer-Supported Cooperative Work and Social Computing).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Luther, K, Tolentino, JL, Wu, W, Pavel, A, Bailey, BP, Agrawala, M, Hartmann, B & Dow, SP 2015, Structuring, aggregating, and evaluating crowdsourced design critique. in CSCW 2015 - Proceedings of the 2015 ACM International Conference on Computer-Supported Cooperative Work and Social Computing. CSCW 2015 - Proceedings of the 2015 ACM International Conference on Computer-Supported Cooperative Work and Social Computing, Association for Computing Machinery, Inc, pp. 473-485, 18th ACM International Conference on Computer-Supported Cooperative Work and Social Computing, CSCW 2015, BC, Canada, 3/14/15. https://doi.org/10.1145/2675133.2675283

Luther K, Tolentino JL, Wu W, Pavel A, Bailey BP, Agrawala M et al. Structuring, aggregating, and evaluating crowdsourced design critique. In CSCW 2015 - Proceedings of the 2015 ACM International Conference on Computer-Supported Cooperative Work and Social Computing. Association for Computing Machinery, Inc. 2015. p. 473-485. (CSCW 2015 - Proceedings of the 2015 ACM International Conference on Computer-Supported Cooperative Work and Social Computing). https://doi.org/10.1145/2675133.2675283

Luther, Kurt ; Tolentino, Jari Lee ; Wu, Wei ; Pavel, Amy ; Bailey, Brian P. ; Agrawala, Maneesh ; Hartmann, Björn ; Dow, Steven P. / Structuring, aggregating, and evaluating crowdsourced design critique. CSCW 2015 - Proceedings of the 2015 ACM International Conference on Computer-Supported Cooperative Work and Social Computing. Association for Computing Machinery, Inc, 2015. pp. 473-485 (CSCW 2015 - Proceedings of the 2015 ACM International Conference on Computer-Supported Cooperative Work and Social Computing).

@inproceedings{df5c51f3c84c4e98bef251e25bb91df9,

title = "Structuring, aggregating, and evaluating crowdsourced design critique",

abstract = "Feedback is an important component of the design process, but gaining access to high-quality critique outside a classroom or firm is challenging. We present CrowdCrit, a webbased system that allows designers to receive design critiques from non-expert crowd workers. We evaluated CrowdCrit in three studies focusing on the designer's experience and benefits of the critiques. In the first study, we compared crowd and expert critiques and found evidence that aggregated crowd critique approaches expert critique. In a second study, we found that designers who got crowd feedback perceived that it improved their design process. The third study showed that designers were enthusiastic about crowd critiques and used them to change their designs. We conclude with implications for the design of crowd feedback services.",

keywords = "Design; critique; feedback; social computing; crowdsourcing",

author = "Kurt Luther and Tolentino, {Jari Lee} and Wei Wu and Amy Pavel and Bailey, {Brian P.} and Maneesh Agrawala and Bj{\"o}rn Hartmann and Dow, {Steven P.}",

note = "Funding Information: National Science Foundation under IIS grants 1210836, 1208382, and 1217096.; 18th ACM International Conference on Computer-Supported Cooperative Work and Social Computing, CSCW 2015 ; Conference date: 14-03-2015 Through 18-03-2015",

year = "2015",

month = feb,

day = "28",

doi = "10.1145/2675133.2675283",

language = "English (US)",

series = "CSCW 2015 - Proceedings of the 2015 ACM International Conference on Computer-Supported Cooperative Work and Social Computing",

publisher = "Association for Computing Machinery, Inc",

pages = "473--485",

booktitle = "CSCW 2015 - Proceedings of the 2015 ACM International Conference on Computer-Supported Cooperative Work and Social Computing",

}

TY - GEN

T1 - Structuring, aggregating, and evaluating crowdsourced design critique

AU - Luther, Kurt

AU - Tolentino, Jari Lee

AU - Wu, Wei

AU - Pavel, Amy

AU - Bailey, Brian P.

AU - Agrawala, Maneesh

AU - Hartmann, Björn

AU - Dow, Steven P.

N1 - Funding Information: National Science Foundation under IIS grants 1210836, 1208382, and 1217096.

PY - 2015/2/28

Y1 - 2015/2/28

N2 - Feedback is an important component of the design process, but gaining access to high-quality critique outside a classroom or firm is challenging. We present CrowdCrit, a webbased system that allows designers to receive design critiques from non-expert crowd workers. We evaluated CrowdCrit in three studies focusing on the designer's experience and benefits of the critiques. In the first study, we compared crowd and expert critiques and found evidence that aggregated crowd critique approaches expert critique. In a second study, we found that designers who got crowd feedback perceived that it improved their design process. The third study showed that designers were enthusiastic about crowd critiques and used them to change their designs. We conclude with implications for the design of crowd feedback services.

AB - Feedback is an important component of the design process, but gaining access to high-quality critique outside a classroom or firm is challenging. We present CrowdCrit, a webbased system that allows designers to receive design critiques from non-expert crowd workers. We evaluated CrowdCrit in three studies focusing on the designer's experience and benefits of the critiques. In the first study, we compared crowd and expert critiques and found evidence that aggregated crowd critique approaches expert critique. In a second study, we found that designers who got crowd feedback perceived that it improved their design process. The third study showed that designers were enthusiastic about crowd critiques and used them to change their designs. We conclude with implications for the design of crowd feedback services.

KW - Design; critique; feedback; social computing; crowdsourcing

UR - http://www.scopus.com/inward/record.url?scp=84968747313&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84968747313&partnerID=8YFLogxK

U2 - 10.1145/2675133.2675283

DO - 10.1145/2675133.2675283

M3 - Conference contribution

AN - SCOPUS:84968747313

T3 - CSCW 2015 - Proceedings of the 2015 ACM International Conference on Computer-Supported Cooperative Work and Social Computing

SP - 473

EP - 485

BT - CSCW 2015 - Proceedings of the 2015 ACM International Conference on Computer-Supported Cooperative Work and Social Computing

PB - Association for Computing Machinery, Inc

T2 - 18th ACM International Conference on Computer-Supported Cooperative Work and Social Computing, CSCW 2015

Y2 - 14 March 2015 through 18 March 2015

ER -