Evaluating the crowd with confidence

Manas Joglekar; Hector Garcia-Molina; Aditya Parameswaran

doi:10.1145/2487575.2487595

Evaluating the crowd with confidence

Manas Joglekar, Hector Garcia-Molina, Aditya Parameswaran

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Worker quality control is a crucial aspect of crowdsourcing systems; typically occupying a large fraction of the time and money invested on crowdsourcing. In this work, we devise techniques to generate confidence intervals for worker error rate estimates, thereby enabling a better evaluation of worker quality. We show that our techniques generate correct confidence intervals on a range of real-world datasets, and demonstrate wide applicability by using them to evict poorly performing workers, and provide confidence intervals on the accuracy of the answers.

Original language	English (US)
Title of host publication	KDD 2013 - 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
Editors	Rajesh Parekh, Jingrui He, Dhillon S. Inderjit, Paul Bradley, Yehuda Koren, Rayid Ghani, Ted E. Senator, Robert L. Grossman, Ramasamy Uthurusamy
Publisher	Association for Computing Machinery
Pages	686-694
Number of pages	9
ISBN (Electronic)	9781450321747
DOIs	https://doi.org/10.1145/2487575.2487595
State	Published - Aug 11 2013
Event	19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2013 - Chicago, United States Duration: Aug 11 2013 → Aug 14 2013

Publication series

Name	Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
Volume	Part F128815

Other

Other	19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2013
Country/Territory	United States
City	Chicago
Period	8/11/13 → 8/14/13

Keywords

Confidence
Crowdsourcing

ASJC Scopus subject areas

Software
Information Systems

Online availability

10.1145/2487575.2487595

Library availability

Discover UIUC Full Text

Cite this

Joglekar, M., Garcia-Molina, H., & Parameswaran, A. (2013). Evaluating the crowd with confidence. In R. Parekh, J. He, D. S. Inderjit, P. Bradley, Y. Koren, R. Ghani, T. E. Senator, R. L. Grossman, & R. Uthurusamy (Eds.), KDD 2013 - 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 686-694). [2487595] (Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; Vol. Part F128815). Association for Computing Machinery. https://doi.org/10.1145/2487575.2487595

Evaluating the crowd with confidence. / Joglekar, Manas; Garcia-Molina, Hector; Parameswaran, Aditya.

KDD 2013 - 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ed. / Rajesh Parekh; Jingrui He; Dhillon S. Inderjit; Paul Bradley; Yehuda Koren; Rayid Ghani; Ted E. Senator; Robert L. Grossman; Ramasamy Uthurusamy. Association for Computing Machinery, 2013. p. 686-694 2487595 (Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; Vol. Part F128815).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Joglekar, M, Garcia-Molina, H & Parameswaran, A 2013, Evaluating the crowd with confidence. in R Parekh, J He, DS Inderjit, P Bradley, Y Koren, R Ghani, TE Senator, RL Grossman & R Uthurusamy (eds), KDD 2013 - 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining., 2487595, Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, vol. Part F128815, Association for Computing Machinery, pp. 686-694, 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2013, Chicago, United States, 8/11/13. https://doi.org/10.1145/2487575.2487595

Joglekar M, Garcia-Molina H, Parameswaran A. Evaluating the crowd with confidence. In Parekh R, He J, Inderjit DS, Bradley P, Koren Y, Ghani R, Senator TE, Grossman RL, Uthurusamy R, editors, KDD 2013 - 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery. 2013. p. 686-694. 2487595. (Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining). doi: 10.1145/2487575.2487595

Joglekar, Manas ; Garcia-Molina, Hector ; Parameswaran, Aditya. / Evaluating the crowd with confidence. KDD 2013 - 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. editor / Rajesh Parekh ; Jingrui He ; Dhillon S. Inderjit ; Paul Bradley ; Yehuda Koren ; Rayid Ghani ; Ted E. Senator ; Robert L. Grossman ; Ramasamy Uthurusamy. Association for Computing Machinery, 2013. pp. 686-694 (Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining).

@inproceedings{2d807aeefbc54ba0bdd4b90d86eec42b,

title = "Evaluating the crowd with confidence",

abstract = "Worker quality control is a crucial aspect of crowdsourcing systems; typically occupying a large fraction of the time and money invested on crowdsourcing. In this work, we devise techniques to generate confidence intervals for worker error rate estimates, thereby enabling a better evaluation of worker quality. We show that our techniques generate correct confidence intervals on a range of real-world datasets, and demonstrate wide applicability by using them to evict poorly performing workers, and provide confidence intervals on the accuracy of the answers.",

keywords = "Confidence, Crowdsourcing",

author = "Manas Joglekar and Hector Garcia-Molina and Aditya Parameswaran",

note = "Publisher Copyright: Copyright {\textcopyright} 2013 ACM. Copyright: Copyright 2018 Elsevier B.V., All rights reserved.; 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2013 ; Conference date: 11-08-2013 Through 14-08-2013",

year = "2013",

month = aug,

day = "11",

doi = "10.1145/2487575.2487595",

language = "English (US)",

series = "Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining",

publisher = "Association for Computing Machinery",

pages = "686--694",

editor = "Rajesh Parekh and Jingrui He and Inderjit, {Dhillon S.} and Paul Bradley and Yehuda Koren and Rayid Ghani and Senator, {Ted E.} and Grossman, {Robert L.} and Ramasamy Uthurusamy",

booktitle = "KDD 2013 - 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining",

}

TY - GEN

T1 - Evaluating the crowd with confidence

AU - Joglekar, Manas

AU - Garcia-Molina, Hector

AU - Parameswaran, Aditya

PY - 2013/8/11

Y1 - 2013/8/11

N2 - Worker quality control is a crucial aspect of crowdsourcing systems; typically occupying a large fraction of the time and money invested on crowdsourcing. In this work, we devise techniques to generate confidence intervals for worker error rate estimates, thereby enabling a better evaluation of worker quality. We show that our techniques generate correct confidence intervals on a range of real-world datasets, and demonstrate wide applicability by using them to evict poorly performing workers, and provide confidence intervals on the accuracy of the answers.

AB - Worker quality control is a crucial aspect of crowdsourcing systems; typically occupying a large fraction of the time and money invested on crowdsourcing. In this work, we devise techniques to generate confidence intervals for worker error rate estimates, thereby enabling a better evaluation of worker quality. We show that our techniques generate correct confidence intervals on a range of real-world datasets, and demonstrate wide applicability by using them to evict poorly performing workers, and provide confidence intervals on the accuracy of the answers.

KW - Confidence

KW - Crowdsourcing

UR - http://www.scopus.com/inward/record.url?scp=84906822235&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84906822235&partnerID=8YFLogxK

U2 - 10.1145/2487575.2487595

DO - 10.1145/2487575.2487595

M3 - Conference contribution

AN - SCOPUS:84906822235

T3 - Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

SP - 686

EP - 694

BT - KDD 2013 - 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

A2 - Parekh, Rajesh

A2 - He, Jingrui

A2 - Inderjit, Dhillon S.

A2 - Bradley, Paul

A2 - Koren, Yehuda

A2 - Ghani, Rayid

A2 - Senator, Ted E.

A2 - Grossman, Robert L.

A2 - Uthurusamy, Ramasamy

PB - Association for Computing Machinery

T2 - 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2013

Y2 - 11 August 2013 through 14 August 2013

ER -

Evaluating the crowd with confidence

Abstract

Publication series

Other

Keywords

ASJC Scopus subject areas

Online availability

Library availability

Related links

Fingerprint

Cite this