Optimal crowd-powered rating and filtering algorithms

We focus on crowd-powered fltering, i.e., fltering a large set of items using humans. Filtering is one of the most commonly used building blocks in crowdsourcing applications and systems. While solutions for crowd-powered fltering exist, theymake a range of implicit assumptions and restrictions, ultimately rendering them not powerful enough for real-world applications. We describe two approaches to discard these implicit assumptions and restrictions: one, that carefully generalizes priorwork, leading to an optimal, but oftentimes intractable solution, and another, that provides a novel way of reasoning about fltering strategies, leading to a sometimes suboptimal, but effciently computable solution (that is asymptotically close to optimal). We demonstrate that our techniques lead to signi ficant reductions in error of up to ì{thorn}Û for fixed cost over prior work in a novel crowdsourcing application: peer evaluation in online courses.