Optimal crowd-powered rating and filtering algorithms

Aditya Parameswaran, Stephen Boyd, Hector Garcia-Molina, Ashish Gupta, Neoklis Polyzotis, Jennifer Widom

Research output: Contribution to journalArticlepeer-review


We focus on crowd-powered fltering, i.e., fltering a large set of items using humans. Filtering is one of the most commonly used building blocks in crowdsourcing applications and systems. While solutions for crowd-powered fltering exist, theymake a range of implicit assumptions and restrictions, ultimately rendering them not powerful enough for real-world applications. We describe two approaches to discard these implicit assumptions and restrictions: one, that carefully generalizes priorwork, leading to an optimal, but oftentimes intractable solution, and another, that provides a novel way of reasoning about fltering strategies, leading to a sometimes suboptimal, but effciently computable solution (that is asymptotically close to optimal). We demonstrate that our techniques lead to signi ficant reductions in error of up to ì{thorn}Û for fixed cost over prior work in a novel crowdsourcing application: peer evaluation in online courses.

Original languageEnglish (US)
Pages (from-to)685-696
Number of pages12
JournalProceedings of the VLDB Endowment
Issue number9
StatePublished - 2014

ASJC Scopus subject areas

  • Computer Science (miscellaneous)
  • Computer Science(all)

Fingerprint Dive into the research topics of 'Optimal crowd-powered rating and filtering algorithms'. Together they form a unique fingerprint.

Cite this