The MIREX grand challenge: A framework of holistic user-experience evaluation in music information retrieval

Xiao Hu, Jin Ha Lee, David Bainbridge, Kahyun Choi, Peter Organisciak, J. Stephen Downie

Research output: Contribution to journalArticlepeer-review

Abstract

Music Information Retrieval (MIR) evaluation has traditionally focused on system-centered approaches where components of MIR systems are evaluated against predefined data sets and golden answers (i.e., ground truth). There are two major limitations of such system-centered evaluation approaches: (a) The evaluation focuses on subtasks in music information retrieval, but not on entire systems and (b) users and their interactions with MIR systems are largely excluded. This article describes the first implementation of a holistic user-experience evaluation in MIR, the MIREX Grand Challenge, where complete MIR systems are evaluated, with user experience being the single overarching goal. It is the first time that complete MIR systems have been evaluated with end users in a realistic scenario. We present the design of the evaluation task, the evaluation criteria and a novel evaluation interface, and the data-collection platform. This is followed by an analysis of the results, reflection on the experience and lessons learned, and plans for future directions.

Original languageEnglish (US)
Pages (from-to)97-112
Number of pages16
JournalJournal of the Association for Information Science and Technology
Volume68
Issue number1
DOIs
StatePublished - Jan 1 2017

Keywords

  • evaluation
  • information retrieval
  • music

ASJC Scopus subject areas

  • Information Systems
  • Computer Networks and Communications
  • Information Systems and Management
  • Library and Information Sciences

Fingerprint

Dive into the research topics of 'The MIREX grand challenge: A framework of holistic user-experience evaluation in music information retrieval'. Together they form a unique fingerprint.

Cite this