Utility data annotation with Amazon Mechanical Turk

Alexander Sorokin, David Alexander Forsyth

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We show how to outsource data annotation to Amazon Mechanical Turk. Doing so has produced annotations in quite large numbers relatively cheaply. The quality is good, and can be checked and controlled. Annotations are produced quickly. We describe results for several different annotation problems. We describe some strategies for determining when the task is well specified and properly priced.

Original languageEnglish (US)
Title of host publication2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops
DOIs
StatePublished - Sep 24 2008
Event2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops - Anchorage, AK, United States
Duration: Jun 23 2008Jun 28 2008

Publication series

Name2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops

Other

Other2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops
CountryUnited States
CityAnchorage, AK
Period6/23/086/28/08

ASJC Scopus subject areas

  • Computer Vision and Pattern Recognition
  • Electrical and Electronic Engineering

Cite this

Sorokin, A., & Forsyth, D. A. (2008). Utility data annotation with Amazon Mechanical Turk. In 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops [4562953] (2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops). https://doi.org/10.1109/CVPRW.2008.4562953

Utility data annotation with Amazon Mechanical Turk. / Sorokin, Alexander; Forsyth, David Alexander.

2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops. 2008. 4562953 (2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Sorokin, A & Forsyth, DA 2008, Utility data annotation with Amazon Mechanical Turk. in 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops., 4562953, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops, Anchorage, AK, United States, 6/23/08. https://doi.org/10.1109/CVPRW.2008.4562953
Sorokin A, Forsyth DA. Utility data annotation with Amazon Mechanical Turk. In 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops. 2008. 4562953. (2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops). https://doi.org/10.1109/CVPRW.2008.4562953
Sorokin, Alexander ; Forsyth, David Alexander. / Utility data annotation with Amazon Mechanical Turk. 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops. 2008. (2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops).
@inproceedings{fa5bfde1abc345a7b98453854dec26f7,
title = "Utility data annotation with Amazon Mechanical Turk",
abstract = "We show how to outsource data annotation to Amazon Mechanical Turk. Doing so has produced annotations in quite large numbers relatively cheaply. The quality is good, and can be checked and controlled. Annotations are produced quickly. We describe results for several different annotation problems. We describe some strategies for determining when the task is well specified and properly priced.",
author = "Alexander Sorokin and Forsyth, {David Alexander}",
year = "2008",
month = "9",
day = "24",
doi = "10.1109/CVPRW.2008.4562953",
language = "English (US)",
isbn = "9781424423408",
series = "2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops",
booktitle = "2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops",

}

TY - GEN

T1 - Utility data annotation with Amazon Mechanical Turk

AU - Sorokin, Alexander

AU - Forsyth, David Alexander

PY - 2008/9/24

Y1 - 2008/9/24

N2 - We show how to outsource data annotation to Amazon Mechanical Turk. Doing so has produced annotations in quite large numbers relatively cheaply. The quality is good, and can be checked and controlled. Annotations are produced quickly. We describe results for several different annotation problems. We describe some strategies for determining when the task is well specified and properly priced.

AB - We show how to outsource data annotation to Amazon Mechanical Turk. Doing so has produced annotations in quite large numbers relatively cheaply. The quality is good, and can be checked and controlled. Annotations are produced quickly. We describe results for several different annotation problems. We describe some strategies for determining when the task is well specified and properly priced.

UR - http://www.scopus.com/inward/record.url?scp=52049123532&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=52049123532&partnerID=8YFLogxK

U2 - 10.1109/CVPRW.2008.4562953

DO - 10.1109/CVPRW.2008.4562953

M3 - Conference contribution

AN - SCOPUS:52049123532

SN - 9781424423408

T3 - 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops

BT - 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops

ER -