Holistic crowd-powered sorting via aiD: Optimizing for accuracies, inconsistencies, and difficulties

Shreya Rajpal, Aditya Parameswaran

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We revisit the fundamental problem of sorting objects using crowdsourced pairwise comparisons. Prior work either treats these comparisons as independent tasks-in which case the resulting judgments may end up being inconsistent, or fails to capture the accuracies of workers, or difficulties of the pairwise comparisons-in which case the resulting judgments may end up being consistent with each other, but ultimately more inaccurate. We adopt a holistic approach that constructs a graph across the set of objects respecting consistency constraints. Our key contribution is a novel method of encoding difficulty of comparisons in the form of constraints on edges. We couple that with an iterative E-M-style procedure to uncover information about latent variables and constraints, along with the graph structure. We show that our approach predicts edge directions as well as difficulty values more accurately than baselines on both real and simulated data, across graphs of various sizes.

Original languageEnglish (US)
Title of host publicationCIKM 2018 - Proceedings of the 27th ACM International Conference on Information and Knowledge Management
EditorsNorman Paton, Selcuk Candan, Haixun Wang, James Allan, Rakesh Agrawal, Alexandros Labrinidis, Alfredo Cuzzocrea, Mohammed Zaki, Divesh Srivastava, Andrei Broder, Assaf Schuster
PublisherAssociation for Computing Machinery
Pages1607-1610
Number of pages4
ISBN (Electronic)9781450360142
DOIs
StatePublished - Oct 17 2018
Event27th ACM International Conference on Information and Knowledge Management, CIKM 2018 - Torino, Italy
Duration: Oct 22 2018Oct 26 2018

Publication series

NameInternational Conference on Information and Knowledge Management, Proceedings

Other

Other27th ACM International Conference on Information and Knowledge Management, CIKM 2018
CountryItaly
CityTorino
Period10/22/1810/26/18

Keywords

  • Constrained optimization
  • Crowdsourcing
  • Graphs
  • Pairwise comparisons

ASJC Scopus subject areas

  • Decision Sciences(all)
  • Business, Management and Accounting(all)

Fingerprint Dive into the research topics of 'Holistic crowd-powered sorting via aiD: Optimizing for accuracies, inconsistencies, and difficulties'. Together they form a unique fingerprint.

  • Cite this

    Rajpal, S., & Parameswaran, A. (2018). Holistic crowd-powered sorting via aiD: Optimizing for accuracies, inconsistencies, and difficulties. In N. Paton, S. Candan, H. Wang, J. Allan, R. Agrawal, A. Labrinidis, A. Cuzzocrea, M. Zaki, D. Srivastava, A. Broder, & A. Schuster (Eds.), CIKM 2018 - Proceedings of the 27th ACM International Conference on Information and Knowledge Management (pp. 1607-1610). (International Conference on Information and Knowledge Management, Proceedings). Association for Computing Machinery. https://doi.org/10.1145/3269206.3269279