Enabling the verification of computational results

Victoria Stodden, Matthew S. Krafczyk, Adhithya Bhaskar

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

The ability to independently regenerate published computational claims is widely recognized as a key component of scientific reproducibility. In this article we take a narrow interpretation of this goal, and attempt to regenerate published claims from author-supplied information, including data, code, inputs, and other provided specifications, on a different computational system than that used by the original authors. We are motivated by Claerbout and Donoho’s exhortation of the importance of providing complete information for reproducibility of the published claim. We chose the Elsevier journal, the Journal of Computational Physics, which has stated author guidelines that encourage the availability of computational digital artifacts that support scholarly findings. In an IRB approved study at the University of Illinois at Urbana-Champaign (IRB #17329) we gathered artifacts from a sample of authors who published in this journal in 2016 and 2017. We then used the ICERM criteria generated at the 2012 ICERM workshop “Reproducibility in Computational and Experimental Mathematics” to evaluate the sufficiency of the information provided in the publications and the ease with which the digital artifacts afforded computational reproducibility. We find that, for the articles for which we obtained computational artifacts, we could not easily regenerate the findings for 67% of them, and we were unable to easily regenerate all the findings for any of the articles. We then evaluated the artifacts we did obtain (55 of 306 articles) and find that the main barriers to computational reproducibility are inadequate documentation of code, data, and workflow information (70.9%), missing code function and setting information, and missing licensing information (75%). We recommend improvements based on these findings, including the deposit of supporting digital artifacts for reproducibility as a condition of publication, and verification of computational findings via re-execution of the code when possible.

Original languageEnglish (US)
Title of host publicationProceedings of the 1st International Workshop on Practical Reproducible Evaluation of Computer Systems, P-RECS 2018
PublisherAssociation for Computing Machinery, Inc
ISBN (Electronic)9781450358613
DOIs
StatePublished - Jun 11 2018
Event1st International Workshop on Practical Reproducible Evaluation of Computer Systems, P-RECS 2018 - Tempe, United States
Duration: Jun 11 2018 → …

Publication series

NameProceedings of the 1st International Workshop on Practical Reproducible Evaluation of Computer Systems, P-RECS 2018

Other

Other1st International Workshop on Practical Reproducible Evaluation of Computer Systems, P-RECS 2018
CountryUnited States
CityTempe
Period6/11/18 → …

Keywords

  • Code access
  • Data access
  • Provenance
  • Reproducibility policy
  • Reproducible research
  • Workflows

ASJC Scopus subject areas

  • Computational Theory and Mathematics
  • Computer Science Applications
  • Software

Fingerprint Dive into the research topics of 'Enabling the verification of computational results'. Together they form a unique fingerprint.

Cite this