An empirical analysis of journal policy effectiveness for computational reproducibility

Victoria Stodden, Jennifer Seiler, Zhaokun Ma

Research output: Contribution to journalArticle

Abstract

A key component of scientific communication is sufficient information for other researchers in the field to reproduce published findings. For computational and data-enabled research, this has often been interpreted to mean making available the raw data from which results were generated, the computer code that generated the findings, and any additional information needed such as workflows and input parameters. Many journals are revising author guidelines to include data and code availability. This work evaluates the effectiveness of journal policy that requires the data and code necessary for reproducibility be made available postpub-lication by the authors upon request. We assess the effectiveness of such a policy by (i) requesting data and code from authors and (ii) attempting replication of the published findings. We chose a random sample of 204 scientific papers published in the journal Science after the implementation of their policy in February 2011. We found that we were able to obtain artifacts from 44% of our sample and were able to reproduce the findings for 26%. We find this policy—author remission of data and code postpublication upon request—an improvement over no policy, but currently insufficient for reproducibility.

Original languageEnglish (US)
Pages (from-to)2584-2589
Number of pages6
JournalProceedings of the National Academy of Sciences of the United States of America
Volume115
Issue number11
DOIs
StatePublished - Mar 13 2018

Fingerprint

workflow
random sample
artifact
communication
science

Keywords

  • Code access
  • Data access
  • Open science
  • Reproducibility policy
  • Reproducible research

ASJC Scopus subject areas

  • General

Cite this

An empirical analysis of journal policy effectiveness for computational reproducibility. / Stodden, Victoria; Seiler, Jennifer; Ma, Zhaokun.

In: Proceedings of the National Academy of Sciences of the United States of America, Vol. 115, No. 11, 13.03.2018, p. 2584-2589.

Research output: Contribution to journalArticle

@article{bbabe62cdef9484abfec7864a9db790b,
title = "An empirical analysis of journal policy effectiveness for computational reproducibility",
abstract = "A key component of scientific communication is sufficient information for other researchers in the field to reproduce published findings. For computational and data-enabled research, this has often been interpreted to mean making available the raw data from which results were generated, the computer code that generated the findings, and any additional information needed such as workflows and input parameters. Many journals are revising author guidelines to include data and code availability. This work evaluates the effectiveness of journal policy that requires the data and code necessary for reproducibility be made available postpub-lication by the authors upon request. We assess the effectiveness of such a policy by (i) requesting data and code from authors and (ii) attempting replication of the published findings. We chose a random sample of 204 scientific papers published in the journal Science after the implementation of their policy in February 2011. We found that we were able to obtain artifacts from 44{\%} of our sample and were able to reproduce the findings for 26{\%}. We find this policy—author remission of data and code postpublication upon request—an improvement over no policy, but currently insufficient for reproducibility.",
keywords = "Code access, Data access, Open science, Reproducibility policy, Reproducible research",
author = "Victoria Stodden and Jennifer Seiler and Zhaokun Ma",
year = "2018",
month = "3",
day = "13",
doi = "10.1073/pnas.1708290115",
language = "English (US)",
volume = "115",
pages = "2584--2589",
journal = "Proceedings of the National Academy of Sciences of the United States of America",
issn = "0027-8424",
number = "11",

}

TY - JOUR

T1 - An empirical analysis of journal policy effectiveness for computational reproducibility

AU - Stodden, Victoria

AU - Seiler, Jennifer

AU - Ma, Zhaokun

PY - 2018/3/13

Y1 - 2018/3/13

N2 - A key component of scientific communication is sufficient information for other researchers in the field to reproduce published findings. For computational and data-enabled research, this has often been interpreted to mean making available the raw data from which results were generated, the computer code that generated the findings, and any additional information needed such as workflows and input parameters. Many journals are revising author guidelines to include data and code availability. This work evaluates the effectiveness of journal policy that requires the data and code necessary for reproducibility be made available postpub-lication by the authors upon request. We assess the effectiveness of such a policy by (i) requesting data and code from authors and (ii) attempting replication of the published findings. We chose a random sample of 204 scientific papers published in the journal Science after the implementation of their policy in February 2011. We found that we were able to obtain artifacts from 44% of our sample and were able to reproduce the findings for 26%. We find this policy—author remission of data and code postpublication upon request—an improvement over no policy, but currently insufficient for reproducibility.

AB - A key component of scientific communication is sufficient information for other researchers in the field to reproduce published findings. For computational and data-enabled research, this has often been interpreted to mean making available the raw data from which results were generated, the computer code that generated the findings, and any additional information needed such as workflows and input parameters. Many journals are revising author guidelines to include data and code availability. This work evaluates the effectiveness of journal policy that requires the data and code necessary for reproducibility be made available postpub-lication by the authors upon request. We assess the effectiveness of such a policy by (i) requesting data and code from authors and (ii) attempting replication of the published findings. We chose a random sample of 204 scientific papers published in the journal Science after the implementation of their policy in February 2011. We found that we were able to obtain artifacts from 44% of our sample and were able to reproduce the findings for 26%. We find this policy—author remission of data and code postpublication upon request—an improvement over no policy, but currently insufficient for reproducibility.

KW - Code access

KW - Data access

KW - Open science

KW - Reproducibility policy

KW - Reproducible research

UR - http://www.scopus.com/inward/record.url?scp=85043778648&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85043778648&partnerID=8YFLogxK

U2 - 10.1073/pnas.1708290115

DO - 10.1073/pnas.1708290115

M3 - Article

VL - 115

SP - 2584

EP - 2589

JO - Proceedings of the National Academy of Sciences of the United States of America

JF - Proceedings of the National Academy of Sciences of the United States of America

SN - 0027-8424

IS - 11

ER -