Distributed estimation via paid crowd work

Song Jianhan, Vei Wang Isaac Phua, Lav R Varshney

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Consider a distributed estimation problem to be carried out by paid crowdworkers, where results are to be returned quickly and accurately. Estimation accuracy is a function of the number of workers completing the job and of the quality of the workers, both of which may be influenced by the payment offered. With limited budget, payment allocation should consider both effects to obtain best results. Since people are not deterministic, payment offers will lead to a random number of variable-quality workers, as governed by choice models. We consider average performance and focus on estimating a parameter from measurements through uniform noise. Since we have shown the optimality of the midrange estimator in specific settings of the general problem, we focus on the best linear unbiased estimator based on order statistics (BLUE-OS) under the mean-squared error (MSE) criterion. Best payment allocations are determined for single crowd platforms, joint population models and separated platform models. Illustrative numerical examples are provided.

Original languageEnglish (US)
Title of host publication2016 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages6200-6204
Number of pages5
ISBN (Electronic)9781479999880
DOIs
StatePublished - May 18 2016
Event41st IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016 - Shanghai, China
Duration: Mar 20 2016Mar 25 2016

Publication series

NameICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings
Volume2016-May
ISSN (Print)1520-6149

Other

Other41st IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016
CountryChina
CityShanghai
Period3/20/163/25/16

Fingerprint

Statistics

Keywords

  • choice models
  • crowdsourcing
  • distributed estimation
  • resource allocation

ASJC Scopus subject areas

  • Software
  • Signal Processing
  • Electrical and Electronic Engineering

Cite this

Jianhan, S., Phua, V. W. I., & Varshney, L. R. (2016). Distributed estimation via paid crowd work. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016 - Proceedings (pp. 6200-6204). [7472869] (ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings; Vol. 2016-May). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICASSP.2016.7472869

Distributed estimation via paid crowd work. / Jianhan, Song; Phua, Vei Wang Isaac; Varshney, Lav R.

2016 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2016. p. 6200-6204 7472869 (ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings; Vol. 2016-May).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Jianhan, S, Phua, VWI & Varshney, LR 2016, Distributed estimation via paid crowd work. in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016 - Proceedings., 7472869, ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, vol. 2016-May, Institute of Electrical and Electronics Engineers Inc., pp. 6200-6204, 41st IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016, Shanghai, China, 3/20/16. https://doi.org/10.1109/ICASSP.2016.7472869
Jianhan S, Phua VWI, Varshney LR. Distributed estimation via paid crowd work. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016 - Proceedings. Institute of Electrical and Electronics Engineers Inc. 2016. p. 6200-6204. 7472869. (ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings). https://doi.org/10.1109/ICASSP.2016.7472869
Jianhan, Song ; Phua, Vei Wang Isaac ; Varshney, Lav R. / Distributed estimation via paid crowd work. 2016 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2016. pp. 6200-6204 (ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings).
@inproceedings{4e5d4def671f4b7fb48c7d062558be37,
title = "Distributed estimation via paid crowd work",
abstract = "Consider a distributed estimation problem to be carried out by paid crowdworkers, where results are to be returned quickly and accurately. Estimation accuracy is a function of the number of workers completing the job and of the quality of the workers, both of which may be influenced by the payment offered. With limited budget, payment allocation should consider both effects to obtain best results. Since people are not deterministic, payment offers will lead to a random number of variable-quality workers, as governed by choice models. We consider average performance and focus on estimating a parameter from measurements through uniform noise. Since we have shown the optimality of the midrange estimator in specific settings of the general problem, we focus on the best linear unbiased estimator based on order statistics (BLUE-OS) under the mean-squared error (MSE) criterion. Best payment allocations are determined for single crowd platforms, joint population models and separated platform models. Illustrative numerical examples are provided.",
keywords = "choice models, crowdsourcing, distributed estimation, resource allocation",
author = "Song Jianhan and Phua, {Vei Wang Isaac} and Varshney, {Lav R}",
year = "2016",
month = "5",
day = "18",
doi = "10.1109/ICASSP.2016.7472869",
language = "English (US)",
series = "ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "6200--6204",
booktitle = "2016 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016 - Proceedings",
address = "United States",

}

TY - GEN

T1 - Distributed estimation via paid crowd work

AU - Jianhan, Song

AU - Phua, Vei Wang Isaac

AU - Varshney, Lav R

PY - 2016/5/18

Y1 - 2016/5/18

N2 - Consider a distributed estimation problem to be carried out by paid crowdworkers, where results are to be returned quickly and accurately. Estimation accuracy is a function of the number of workers completing the job and of the quality of the workers, both of which may be influenced by the payment offered. With limited budget, payment allocation should consider both effects to obtain best results. Since people are not deterministic, payment offers will lead to a random number of variable-quality workers, as governed by choice models. We consider average performance and focus on estimating a parameter from measurements through uniform noise. Since we have shown the optimality of the midrange estimator in specific settings of the general problem, we focus on the best linear unbiased estimator based on order statistics (BLUE-OS) under the mean-squared error (MSE) criterion. Best payment allocations are determined for single crowd platforms, joint population models and separated platform models. Illustrative numerical examples are provided.

AB - Consider a distributed estimation problem to be carried out by paid crowdworkers, where results are to be returned quickly and accurately. Estimation accuracy is a function of the number of workers completing the job and of the quality of the workers, both of which may be influenced by the payment offered. With limited budget, payment allocation should consider both effects to obtain best results. Since people are not deterministic, payment offers will lead to a random number of variable-quality workers, as governed by choice models. We consider average performance and focus on estimating a parameter from measurements through uniform noise. Since we have shown the optimality of the midrange estimator in specific settings of the general problem, we focus on the best linear unbiased estimator based on order statistics (BLUE-OS) under the mean-squared error (MSE) criterion. Best payment allocations are determined for single crowd platforms, joint population models and separated platform models. Illustrative numerical examples are provided.

KW - choice models

KW - crowdsourcing

KW - distributed estimation

KW - resource allocation

UR - http://www.scopus.com/inward/record.url?scp=84973400164&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84973400164&partnerID=8YFLogxK

U2 - 10.1109/ICASSP.2016.7472869

DO - 10.1109/ICASSP.2016.7472869

M3 - Conference contribution

AN - SCOPUS:84973400164

T3 - ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

SP - 6200

EP - 6204

BT - 2016 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016 - Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

ER -