TY - JOUR
T1 - Capturing the workflows of music information retrieval for repeatability and reuse
AU - Page, Kevin R.
AU - Fields, Ben
AU - Roure, David De
AU - Crawford, Tim
AU - Downie, J. Stephen
N1 - Funding Information:
Acknowledgements This work was carried out through the EPSRC funded e-Research South platform grant (Grant No. EP/F05811X/1), the EU FP7 (ICT-2009.4.1) funded Wf4Ever project (270129), and the Structural Analysis of Large Amounts of Music Information (SALAMI) project funded by the JISC Digitisation and e-Content programme and the National Science Foundation (Grant Nos. IIS 10-42727 and IIS 09-39253) and the Canadian Social Science and Humanities Research Council (SSHRC). We are extremely grateful to Kahyun Choi and other members of the International Music Information Retrieval Systems Evaluation Laboratory (IMIRSEL) at the University of Illinois for their assistance collating the MIREX programming language statistics, and the reviewers of earlier iterations of this paper for their invaluable comments.
PY - 2013/12
Y1 - 2013/12
N2 - Many solutions for the reuse and re-purposing of Music Information Retrieval (MIR) methods, and the tools implementing those methods, have been introduced over recent years. Proposals for achieving interoperability between systems have ranged from shared software libraries and interfaces, through common frameworks and portals, to standardised file formats and metadata. Here we assess these solutions for their suitability to be reused and combined as repurposable components within assemblies (or workflows) that can be used in novel and possibly more ambitious ways. Reuse and repeatability also have great implications for the process of MIR research: the encapsulation of any algorithm and its operation - including inputs, parameters, and outputs - is fundamental to the repeatability and reproducibility of an experiment. This is desirable both for the open and reliable evaluation of algorithms and for the advancement of MIR by building more effectively upon prior research. At present there is no clear best practice widely adopted by the field. Based upon our analysis of contemporary systems and their adoption we reflect as to whether this should be considered a failure. Are there limits to interoperability unique to MIR, and how might they be overcome Beyond workflows how much research context can, and should, be captured We frame our assessment within the emerging notion of Research Objects for reproducible research in other domains, and describe how their adoption could serve as a route to reuse in MIR.
AB - Many solutions for the reuse and re-purposing of Music Information Retrieval (MIR) methods, and the tools implementing those methods, have been introduced over recent years. Proposals for achieving interoperability between systems have ranged from shared software libraries and interfaces, through common frameworks and portals, to standardised file formats and metadata. Here we assess these solutions for their suitability to be reused and combined as repurposable components within assemblies (or workflows) that can be used in novel and possibly more ambitious ways. Reuse and repeatability also have great implications for the process of MIR research: the encapsulation of any algorithm and its operation - including inputs, parameters, and outputs - is fundamental to the repeatability and reproducibility of an experiment. This is desirable both for the open and reliable evaluation of algorithms and for the advancement of MIR by building more effectively upon prior research. At present there is no clear best practice widely adopted by the field. Based upon our analysis of contemporary systems and their adoption we reflect as to whether this should be considered a failure. Are there limits to interoperability unique to MIR, and how might they be overcome Beyond workflows how much research context can, and should, be captured We frame our assessment within the emerging notion of Research Objects for reproducible research in other domains, and describe how their adoption could serve as a route to reuse in MIR.
KW - Music Information Retrieval (MIR)
KW - Reproducible research
KW - Research Objects
KW - Workflows
UR - http://www.scopus.com/inward/record.url?scp=84888343387&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84888343387&partnerID=8YFLogxK
U2 - 10.1007/s10844-013-0260-9
DO - 10.1007/s10844-013-0260-9
M3 - Article
AN - SCOPUS:84888343387
SN - 0925-9902
VL - 41
SP - 435
EP - 459
JO - Journal of Intelligent Information Systems
JF - Journal of Intelligent Information Systems
IS - 3
ER -