TY - GEN
T1 - MIREX grand challenge 2014 user experience
T2 - 16th International Society for Music Information Retrieval Conference, ISMIR 2015
AU - Lee, Jin Ha
AU - Hu, Xiao
AU - Choi, Kahyun
AU - Downie, J. Stephen
N1 - Publisher Copyright:
© Jin Ha Lee, Xiao Hu, Kahyun Choi, J. Stephen Downie.
PY - 2015
Y1 - 2015
N2 - Evaluation has always been fundamental to the Music Information Retrieval (MIR) community, as evidenced by the popularity of the Music Information Retrieval Evaluation eXchange (MIREX). However, prior MIREX tasks have primarily focused on testing specialized MIR algorithms that sit on the back end of systems. Not until the Grand Challenge 2014 User Experience (GC14UX) task had the users’ overall interaction and experience with complete systems been formally evaluated. Three systems were evaluated based on five criteria. This paper reports the results of GC14UX, with a special focus on the qualitative analysis of 99 free text responses collected from evaluators. The analysis revealed additional user opinions, not fully captured by score ratings on the given criteria, and demonstrated the challenge of evaluating a variety of systems with different user goals. We conclude with a discussion on the implications of findings and recommendations for future UX evaluation tasks, including adding new criteria: Aesthetics, Performance, and Utility.
AB - Evaluation has always been fundamental to the Music Information Retrieval (MIR) community, as evidenced by the popularity of the Music Information Retrieval Evaluation eXchange (MIREX). However, prior MIREX tasks have primarily focused on testing specialized MIR algorithms that sit on the back end of systems. Not until the Grand Challenge 2014 User Experience (GC14UX) task had the users’ overall interaction and experience with complete systems been formally evaluated. Three systems were evaluated based on five criteria. This paper reports the results of GC14UX, with a special focus on the qualitative analysis of 99 free text responses collected from evaluators. The analysis revealed additional user opinions, not fully captured by score ratings on the given criteria, and demonstrated the challenge of evaluating a variety of systems with different user goals. We conclude with a discussion on the implications of findings and recommendations for future UX evaluation tasks, including adding new criteria: Aesthetics, Performance, and Utility.
UR - http://www.scopus.com/inward/record.url?scp=85059545495&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85059545495&partnerID=8YFLogxK
M3 - Conference contribution
AN - SCOPUS:85059545495
T3 - Proceedings of the 16th International Society for Music Information Retrieval Conference, ISMIR 2015
SP - 779
EP - 785
BT - Proceedings of the 16th International Society for Music Information Retrieval Conference, ISMIR 2015
A2 - Muller, Meinard
A2 - Wiering, Frans
PB - International Society for Music Information Retrieval
Y2 - 26 October 2015 through 30 October 2015
ER -