TY - GEN
T1 - Systematic analysis of output agreement games
T2 - 2012 AAAI Workshop
AU - Huang, Shih Wen
AU - Fu, Wai-Tat
PY - 2012
Y1 - 2012
N2 - We report results from a human computation study that tests the extent to which output agreement games are better than traditional methods in terms of increasing quality of labels and motivation of voluntary workers on a task with a gold standard. We built an output agreement game that let workers recruited from Amazon's Mechanical Turks label the semantic textual similarity of 20 sentence pairs. To compare and test the effects of the major components of the game, we created interfaces that had different combinations of a gaming environment (G), social interaction (S), and feedback (F). Our results show that the main reason that an output agreement game can collect more high-quality labels is the gaming environment (scoring system, leaderboard, etc). On the other hand, a worker is much more motivated to voluntarily do the task if he or she can do it with another worker (i.e., with social interaction). Our analysis provides human computation researchers important insight on understanding how and why the method of Game with a Purpose (GWAP) can generate high-quality outcomes and motivate more voluntary workers.
AB - We report results from a human computation study that tests the extent to which output agreement games are better than traditional methods in terms of increasing quality of labels and motivation of voluntary workers on a task with a gold standard. We built an output agreement game that let workers recruited from Amazon's Mechanical Turks label the semantic textual similarity of 20 sentence pairs. To compare and test the effects of the major components of the game, we created interfaces that had different combinations of a gaming environment (G), social interaction (S), and feedback (F). Our results show that the main reason that an output agreement game can collect more high-quality labels is the gaming environment (scoring system, leaderboard, etc). On the other hand, a worker is much more motivated to voluntarily do the task if he or she can do it with another worker (i.e., with social interaction). Our analysis provides human computation researchers important insight on understanding how and why the method of Game with a Purpose (GWAP) can generate high-quality outcomes and motivate more voluntary workers.
UR - http://www.scopus.com/inward/record.url?scp=84875733702&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84875733702&partnerID=8YFLogxK
M3 - Conference contribution
AN - SCOPUS:84875733702
SN - 9781577355731
T3 - AAAI Workshop - Technical Report
SP - 2
EP - 8
BT - Human Computation - Papers from the 2012 AAAI Workshop, Technical Report
Y2 - 23 July 2012 through 23 July 2012
ER -