TY - GEN
T1 - Evaluation of algorithms using games
T2 - 10th International Society for Music Information Retrieval Conference, ISMIR 2009
AU - Law, Edith
AU - West, Kris
AU - Mandel, Michael
AU - Bay, Mert
AU - Downie, J. Stephen
PY - 2009
Y1 - 2009
N2 - Search by keyword is an extremely popular method for retrieving music. To support this, novel algorithms that automatically tag music are being developed. The conventional way to evaluate audio tagging algorithms is to compute measures of agreement between the output and the ground truth set. In this work, we introduce a new method for evaluating audio tagging algorithms on a large scale by collecting set-level judgments from players of a human computation game called TagATune. We present the design and preliminary results of an experiment comparing five algorithms using this new evaluation metric, and contrast the results with those obtained by applying several conventional agreement-based evaluation metrics.
AB - Search by keyword is an extremely popular method for retrieving music. To support this, novel algorithms that automatically tag music are being developed. The conventional way to evaluate audio tagging algorithms is to compute measures of agreement between the output and the ground truth set. In this work, we introduce a new method for evaluating audio tagging algorithms on a large scale by collecting set-level judgments from players of a human computation game called TagATune. We present the design and preliminary results of an experiment comparing five algorithms using this new evaluation metric, and contrast the results with those obtained by applying several conventional agreement-based evaluation metrics.
UR - https://www.scopus.com/pages/publications/84873647698
UR - https://www.scopus.com/pages/publications/84873647698#tab=citedBy
M3 - Conference contribution
AN - SCOPUS:84873647698
SN - 9780981353708
T3 - Proceedings of the 10th International Society for Music Information Retrieval Conference, ISMIR 2009
SP - 387
EP - 392
BT - Proceedings of the 10th International Society for Music Information Retrieval Conference, ISMIR 2009
Y2 - 26 October 2009 through 30 October 2009
ER -