Evaluation of algorithms using games: The case of music tagging

Edith Law, Kris West, Michael Mandel, Mert Bay, J Stephen Downie

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Search by keyword is an extremely popular method for retrieving music. To support this, novel algorithms that automatically tag music are being developed. The conventional way to evaluate audio tagging algorithms is to compute measures of agreement between the output and the ground truth set. In this work, we introduce a new method for evaluating audio tagging algorithms on a large scale by collecting set-level judgments from players of a human computation game called TagATune. We present the design and preliminary results of an experiment comparing five algorithms using this new evaluation metric, and contrast the results with those obtained by applying several conventional agreement-based evaluation metrics.

Original languageEnglish (US)
Title of host publicationProceedings of the 10th International Society for Music Information Retrieval Conference, ISMIR 2009
Pages387-392
Number of pages6
StatePublished - 2009
Externally publishedYes
Event10th International Society for Music Information Retrieval Conference, ISMIR 2009 - Kobe, Japan
Duration: Oct 26 2009Oct 30 2009

Other

Other10th International Society for Music Information Retrieval Conference, ISMIR 2009
CountryJapan
CityKobe
Period10/26/0910/30/09

ASJC Scopus subject areas

  • Music
  • Information Systems

Fingerprint Dive into the research topics of 'Evaluation of algorithms using games: The case of music tagging'. Together they form a unique fingerprint.

  • Cite this

    Law, E., West, K., Mandel, M., Bay, M., & Downie, J. S. (2009). Evaluation of algorithms using games: The case of music tagging. In Proceedings of the 10th International Society for Music Information Retrieval Conference, ISMIR 2009 (pp. 387-392)