TY - GEN
T1 - Holistic crowd-powered sorting via aiD
T2 - 27th ACM International Conference on Information and Knowledge Management, CIKM 2018
AU - Rajpal, Shreya
AU - Parameswaran, Aditya
N1 - Funding Information:
We presented a method for determining partial sort orders for a set of items. Our main contributions are a novel method of performing EM that uses information from the graph structure, as well as a cycle breaking algorithm. We showed that our method outperformed baselines on prior work on all metrics. Acknowledgments. We acknowledge support from NSF grant IIS-1652750 and grant W911NF-18-1-0335 awarded by the ARO. The content is the responsibility of the authors and does not necessarily represent the official views of the funding agencies.
Funding Information:
We acknowledge support from NSF grant IIS-1652750 and grant W911NF-18-1-0335 awarded by the ARO. The content is the responsibility of the authors and does not necessarily represent the official views of the funding agencies.
Publisher Copyright:
© 2018 Copyright held by the owner/author(s).
PY - 2018/10/17
Y1 - 2018/10/17
N2 - We revisit the fundamental problem of sorting objects using crowdsourced pairwise comparisons. Prior work either treats these comparisons as independent tasks-in which case the resulting judgments may end up being inconsistent, or fails to capture the accuracies of workers, or difficulties of the pairwise comparisons-in which case the resulting judgments may end up being consistent with each other, but ultimately more inaccurate. We adopt a holistic approach that constructs a graph across the set of objects respecting consistency constraints. Our key contribution is a novel method of encoding difficulty of comparisons in the form of constraints on edges. We couple that with an iterative E-M-style procedure to uncover information about latent variables and constraints, along with the graph structure. We show that our approach predicts edge directions as well as difficulty values more accurately than baselines on both real and simulated data, across graphs of various sizes.
AB - We revisit the fundamental problem of sorting objects using crowdsourced pairwise comparisons. Prior work either treats these comparisons as independent tasks-in which case the resulting judgments may end up being inconsistent, or fails to capture the accuracies of workers, or difficulties of the pairwise comparisons-in which case the resulting judgments may end up being consistent with each other, but ultimately more inaccurate. We adopt a holistic approach that constructs a graph across the set of objects respecting consistency constraints. Our key contribution is a novel method of encoding difficulty of comparisons in the form of constraints on edges. We couple that with an iterative E-M-style procedure to uncover information about latent variables and constraints, along with the graph structure. We show that our approach predicts edge directions as well as difficulty values more accurately than baselines on both real and simulated data, across graphs of various sizes.
KW - Constrained optimization
KW - Crowdsourcing
KW - Graphs
KW - Pairwise comparisons
UR - http://www.scopus.com/inward/record.url?scp=85058051111&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85058051111&partnerID=8YFLogxK
U2 - 10.1145/3269206.3269279
DO - 10.1145/3269206.3269279
M3 - Conference contribution
AN - SCOPUS:85058051111
T3 - International Conference on Information and Knowledge Management, Proceedings
SP - 1607
EP - 1610
BT - CIKM 2018 - Proceedings of the 27th ACM International Conference on Information and Knowledge Management
A2 - Paton, Norman
A2 - Candan, Selcuk
A2 - Wang, Haixun
A2 - Allan, James
A2 - Agrawal, Rakesh
A2 - Labrinidis, Alexandros
A2 - Cuzzocrea, Alfredo
A2 - Zaki, Mohammed
A2 - Srivastava, Divesh
A2 - Broder, Andrei
A2 - Schuster, Assaf
PB - Association for Computing Machinery
Y2 - 22 October 2018 through 26 October 2018
ER -