Abstract
Search by keyword is an extremely popular method for retrieving music. To support this, novel algorithms that automatically tag music are being developed. The conventional way to evaluate audio tagging algorithms is to compute measures of agreement between the output and the ground truth set. In this work, we introduce a new method for evaluating audio tagging algorithms on a large scale by collecting set-level judgments from players of a human computation game called TagATune. We present the design and preliminary results of an experiment comparing five algorithms using this new evaluation metric, and contrast the results with those obtained by applying several conventional agreement-based evaluation metrics.
| Original language | English (US) |
|---|---|
| Title of host publication | Proceedings of the 10th International Society for Music Information Retrieval Conference, ISMIR 2009 |
| Pages | 387-392 |
| Number of pages | 6 |
| State | Published - 2009 |
| Externally published | Yes |
| Event | 10th International Society for Music Information Retrieval Conference, ISMIR 2009 - Kobe, Japan Duration: Oct 26 2009 → Oct 30 2009 |
Other
| Other | 10th International Society for Music Information Retrieval Conference, ISMIR 2009 |
|---|---|
| Country | Japan |
| City | Kobe |
| Period | 10/26/09 → 10/30/09 |
ASJC Scopus subject areas
- Music
- Information Systems
Fingerprint Dive into the research topics of 'Evaluation of algorithms using games: The case of music tagging'. Together they form a unique fingerprint.
Cite this
Law, E., West, K., Mandel, M., Bay, M., & Downie, J. S. (2009). Evaluation of algorithms using games: The case of music tagging. In Proceedings of the 10th International Society for Music Information Retrieval Conference, ISMIR 2009 (pp. 387-392)