Mirex grand challenge 2014 user experience: Qualitative analysis of user feedback

Jin Ha Lee, Xiao Hu, Kahyun Choi, J. Stephen Downie

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Evaluation has always been fundamental to the Music Information Retrieval (MIR) community, as evidenced by the popularity of the Music Information Retrieval Evaluation eXchange (MIREX). However, prior MIREX tasks have primarily focused on testing specialized MIR algorithms that sit on the back end of systems. Not until the Grand Challenge 2014 User Experience (GC14UX) task had the users’ overall interaction and experience with complete systems been formally evaluated. Three systems were evaluated based on five criteria. This paper reports the results of GC14UX, with a special focus on the qualitative analysis of 99 free text responses collected from evaluators. The analysis revealed additional user opinions, not fully captured by score ratings on the given criteria, and demonstrated the challenge of evaluating a variety of systems with different user goals. We conclude with a discussion on the implications of findings and recommendations for future UX evaluation tasks, including adding new criteria: Aesthetics, Performance, and Utility.

Original languageEnglish (US)
Title of host publicationProceedings of the 16th International Society for Music Information Retrieval Conference, ISMIR 2015
EditorsMeinard Muller, Frans Wiering
PublisherInternational Society for Music Information Retrieval
Pages779-785
Number of pages7
ISBN (Electronic)9788460688532
StatePublished - Jan 1 2015
Event16th International Society for Music Information Retrieval Conference, ISMIR 2015 - Malaga, Spain
Duration: Oct 26 2015Oct 30 2015

Publication series

NameProceedings of the 16th International Society for Music Information Retrieval Conference, ISMIR 2015

Conference

Conference16th International Society for Music Information Retrieval Conference, ISMIR 2015
CountrySpain
CityMalaga
Period10/26/1510/30/15

ASJC Scopus subject areas

  • Music
  • Information Systems

Fingerprint Dive into the research topics of 'Mirex grand challenge 2014 user experience: Qualitative analysis of user feedback'. Together they form a unique fingerprint.

  • Cite this

    Lee, J. H., Hu, X., Choi, K., & Downie, J. S. (2015). Mirex grand challenge 2014 user experience: Qualitative analysis of user feedback. In M. Muller, & F. Wiering (Eds.), Proceedings of the 16th International Society for Music Information Retrieval Conference, ISMIR 2015 (pp. 779-785). (Proceedings of the 16th International Society for Music Information Retrieval Conference, ISMIR 2015). International Society for Music Information Retrieval.