Systematic analysis of output agreement games: Effects of gaming environment, social interaction, and feedback

Shih Wen Huang, Wai Tat Fu

Research output: Chapter in Book/Report/Conference proceedingConference contribution


We report results from a human computation study that tests the extent to which output agreement games are better than traditional methods in terms of increasing quality of labels and motivation of voluntary workers on a task with a gold standard. We built an output agreement game that let workers recruited from Amazon's Mechanical Turks label the semantic textual similarity of 20 sentence pairs. To compare and test the effects of the major components of the game, we created interfaces that had different combinations of a gaming environment (G), social interaction (S), and feedback (F). Our results show that the main reason that an output agreement game can collect more high-quality labels is the gaming environment (scoring system, leaderboard, etc). On the other hand, a worker is much more motivated to voluntarily do the task if he or she can do it with another worker (i.e., with social interaction). Our analysis provides human computation researchers important insight on understanding how and why the method of Game with a Purpose (GWAP) can generate high-quality outcomes and motivate more voluntary workers.

Original languageEnglish (US)
Title of host publicationHuman Computation - Papers from the 2012 AAAI Workshop, Technical Report
Number of pages7
StatePublished - 2012
Externally publishedYes
Event2012 AAAI Workshop - Toronto, ON, Canada
Duration: Jul 23 2012Jul 23 2012

Publication series

NameAAAI Workshop - Technical Report


Other2012 AAAI Workshop
CityToronto, ON

ASJC Scopus subject areas

  • General Engineering


Dive into the research topics of 'Systematic analysis of output agreement games: Effects of gaming environment, social interaction, and feedback'. Together they form a unique fingerprint.

Cite this