TY - GEN
T1 - Evaluation of algorithms using games
T2 - 10th International Society for Music Information Retrieval Conference, ISMIR 2009
AU - Law, Edith
AU - West, Kris
AU - Mandel, Michael
AU - Bay, Mert
AU - Stephen Downie, J.
PY - 2009
Y1 - 2009
N2 - Search by keyword is an extremely popular method for retrieving music. To support this, novel algorithms that automatically tag music are being developed. The conventional way to evaluate audio tagging algorithms is to compute measures of agreement between the output and the ground truth set. In this work, we introduce a new method for evaluating audio tagging algorithms on a large scale by collecting set-level judgments from players of a human computation game called TagATune. We present the design and preliminary results of an experiment comparing five algorithms using this new evaluation metric, and contrast the results with those obtained by applying several conventional agreement-based evaluation metrics.
AB - Search by keyword is an extremely popular method for retrieving music. To support this, novel algorithms that automatically tag music are being developed. The conventional way to evaluate audio tagging algorithms is to compute measures of agreement between the output and the ground truth set. In this work, we introduce a new method for evaluating audio tagging algorithms on a large scale by collecting set-level judgments from players of a human computation game called TagATune. We present the design and preliminary results of an experiment comparing five algorithms using this new evaluation metric, and contrast the results with those obtained by applying several conventional agreement-based evaluation metrics.
UR - http://www.scopus.com/inward/record.url?scp=84873647698&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84873647698&partnerID=8YFLogxK
M3 - Conference contribution
AN - SCOPUS:84873647698
SN - 9780981353708
T3 - Proceedings of the 10th International Society for Music Information Retrieval Conference, ISMIR 2009
SP - 387
EP - 392
BT - Proceedings of the 10th International Society for Music Information Retrieval Conference, ISMIR 2009
Y2 - 26 October 2009 through 30 October 2009
ER -