Novices who focused or experts who didn't? How effort and expertise cues affect judgments of crowd work

Y. Wayne Wu, Brian P. Bailey

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Crowd feedback services offer a new method for acquiring feedback during design. A key problem is that the services only return the feedback without any cues about the people who provided it. In this paper, we investigate two cues of a feedback provider - the effort invested in a feedback task and expertise in the domain. First, we tested how positive and negative cues of a provider's effort and expertise affected perceived quality of the feedback. Results showed both cues affected perceived quality, but primarily when the cues were negative. The results also showed that effort cues affected perceived quality as much as expertise. In a second study, we explored the use of behavioral data for modeling effort for feedback tasks. For a binary classification, the models achieved up to 92% accuracy relative to human raters. This result validates the feasibility of implementing effort cues in crowd services. The contributions of this work will enable increased transparency in crowd feedback services, benefiting both designers and feedback providers.

Original languageEnglish (US)
Title of host publicationCHI 2016 - Proceedings, 34th Annual CHI Conference on Human Factors in Computing Systems
PublisherAssociation for Computing Machinery
Pages4086-4097
Number of pages12
ISBN (Electronic)9781450333627
DOIs
StatePublished - May 7 2016
Event34th Annual Conference on Human Factors in Computing Systems, CHI 2016 - San Jose, United States
Duration: May 7 2016May 12 2016

Publication series

NameConference on Human Factors in Computing Systems - Proceedings

Other

Other34th Annual Conference on Human Factors in Computing Systems, CHI 2016
CountryUnited States
CitySan Jose
Period5/7/165/12/16

    Fingerprint

Keywords

  • Creativity
  • Crowdsourcing
  • Design
  • Feedback

ASJC Scopus subject areas

  • Software
  • Human-Computer Interaction
  • Computer Graphics and Computer-Aided Design

Cite this

Wu, Y. W., & Bailey, B. P. (2016). Novices who focused or experts who didn't? How effort and expertise cues affect judgments of crowd work. In CHI 2016 - Proceedings, 34th Annual CHI Conference on Human Factors in Computing Systems (pp. 4086-4097). (Conference on Human Factors in Computing Systems - Proceedings). Association for Computing Machinery. https://doi.org/10.1145/2858036.2858330