Subjective evidence evaluation survey for many-analysts studies

Alexandra Sarafoglou; Suzanne Hoogeveen; Don Van Den Bergh; Balazs Aczel; Casper J. Albers; Tim Althoff; Rotem Botvinik-Nezer; Niko A. Busch; Andrea M. Cataldo; Berna Devezer; Noah N.N. Van Dongen; Anna Dreber; Eiko I. Fried; Rink Hoekstra; Sabine Hoffman; Felix Holzmeister; Jürgen Huber; Nick Huntington-Klein; John Ioannidis; Magnus Johannesson; Michael Kirchler; Eric Loken; Jan Francois Mangin; Dora Matzke; Albert J. Menkveld; Gustav Nilsonne; Don Van Ravenzwaaij; Martin Schweinsberg; Hannah Schulz-Kuempel; David R. Shanks; Daniel J. Simons; Barbara A. Spellman; Andrea H. Stoevenbelt; Barnabas Szaszi; Darinka Trübutschek; Francis Tuerlinckx; Eric L. Uhlmann; Wolf Vanpaemel; Jelte Wicherts; Eric Jan Wagenmakers

doi:10.1098/rsos.240125

Subjective evidence evaluation survey for many-analysts studies

Alexandra Sarafoglou, Suzanne Hoogeveen, Don Van Den Bergh, Balazs Aczel, Casper J. Albers, Tim Althoff, Rotem Botvinik-Nezer, Niko A. Busch, Andrea M. Cataldo, Berna Devezer, Noah N.N. Van Dongen, Anna Dreber, Eiko I. Fried, Rink Hoekstra, Sabine Hoffman, Felix Holzmeister, Jürgen Huber, Nick Huntington-Klein, John Ioannidis, Magnus JohannessonMichael Kirchler, Eric Loken, Jan Francois Mangin, Dora Matzke, Albert J. Menkveld, Gustav Nilsonne, Don Van Ravenzwaaij, Martin Schweinsberg, Hannah Schulz-Kuempel, David R. Shanks, Daniel J. Simons, Barbara A. Spellman, Andrea H. Stoevenbelt, Barnabas Szaszi, Darinka Trübutschek, Francis Tuerlinckx, Eric L. Uhlmann, Wolf Vanpaemel, Jelte Wicherts, Eric Jan Wagenmakers

Research output: Contribution to journal › Article › peer-review

Abstract

Many-analysts studies explore how well an empirical claim withstands plausible alternative analyses of the same dataset by multiple, independent analysis teams. Conclusions from these studies typically rely on a single outcome metric (e.g. effect size) provided by each analysis team. Although informative about the range of plausible effects in a dataset, a single effect size from each team does not provide a complete, nuanced understanding of how analysis choices are related to the outcome. We used the Delphi consensus technique with input from 37 experts to develop an 18-item subjective evidence evaluation survey (SEES) to evaluate how each analysis team views the methodological appropriateness of the research design and the strength of evidence for the hypothesis. We illustrate the usefulness of the SEES in providing richer evidence assessment with pilot data from a previous many-analysts study.

Original language	English (US)
Article number	240125
Journal	Royal Society Open Science
Volume	11
Issue number	7
DOIs	https://doi.org/10.1098/rsos.240125
State	Published - Jul 24 2024

Keywords

crowdsourcing analysis
metascience
open science
scientific transparency
team science

ASJC Scopus subject areas

General

Online availability

10.1098/rsos.240125

Library availability

Discover UIUC Full Text

Cite this

Sarafoglou, A., Hoogeveen, S., Van Den Bergh, D., Aczel, B., Albers, C. J., Althoff, T., Botvinik-Nezer, R., Busch, N. A., Cataldo, A. M., Devezer, B., Van Dongen, N. N. N., Dreber, A., Fried, E. I., Hoekstra, R., Hoffman, S., Holzmeister, F., Huber, J., Huntington-Klein, N., Ioannidis, J., ... Wagenmakers, E. J. (2024). Subjective evidence evaluation survey for many-analysts studies. Royal Society Open Science, 11(7), Article 240125. https://doi.org/10.1098/rsos.240125

Sarafoglou, A, Hoogeveen, S, Van Den Bergh, D, Aczel, B, Albers, CJ, Althoff, T, Botvinik-Nezer, R, Busch, NA, Cataldo, AM, Devezer, B, Van Dongen, NNN, Dreber, A, Fried, EI, Hoekstra, R, Hoffman, S, Holzmeister, F, Huber, J, Huntington-Klein, N, Ioannidis, J, Johannesson, M, Kirchler, M, Loken, E, Mangin, JF, Matzke, D, Menkveld, AJ, Nilsonne, G, Van Ravenzwaaij, D, Schweinsberg, M, Schulz-Kuempel, H, Shanks, DR, Simons, DJ, Spellman, BA, Stoevenbelt, AH, Szaszi, B, Trübutschek, D, Tuerlinckx, F, Uhlmann, EL, Vanpaemel, W, Wicherts, J & Wagenmakers, EJ 2024, 'Subjective evidence evaluation survey for many-analysts studies', Royal Society Open Science, vol. 11, no. 7, 240125. https://doi.org/10.1098/rsos.240125

@article{0a6b00dce43440929a356c2175cad497,

title = "Subjective evidence evaluation survey for many-analysts studies",

abstract = "Many-analysts studies explore how well an empirical claim withstands plausible alternative analyses of the same dataset by multiple, independent analysis teams. Conclusions from these studies typically rely on a single outcome metric (e.g. effect size) provided by each analysis team. Although informative about the range of plausible effects in a dataset, a single effect size from each team does not provide a complete, nuanced understanding of how analysis choices are related to the outcome. We used the Delphi consensus technique with input from 37 experts to develop an 18-item subjective evidence evaluation survey (SEES) to evaluate how each analysis team views the methodological appropriateness of the research design and the strength of evidence for the hypothesis. We illustrate the usefulness of the SEES in providing richer evidence assessment with pilot data from a previous many-analysts study.",

keywords = "crowdsourcing analysis, metascience, open science, scientific transparency, team science",

author = "Alexandra Sarafoglou and Suzanne Hoogeveen and {Van Den Bergh}, Don and Balazs Aczel and Albers, {Casper J.} and Tim Althoff and Rotem Botvinik-Nezer and Busch, {Niko A.} and Cataldo, {Andrea M.} and Berna Devezer and {Van Dongen}, {Noah N.N.} and Anna Dreber and Fried, {Eiko I.} and Rink Hoekstra and Sabine Hoffman and Felix Holzmeister and J{\"u}rgen Huber and Nick Huntington-Klein and John Ioannidis and Magnus Johannesson and Michael Kirchler and Eric Loken and Mangin, {Jan Francois} and Dora Matzke and Menkveld, {Albert J.} and Gustav Nilsonne and {Van Ravenzwaaij}, Don and Martin Schweinsberg and Hannah Schulz-Kuempel and Shanks, {David R.} and Simons, {Daniel J.} and Spellman, {Barbara A.} and Stoevenbelt, {Andrea H.} and Barnabas Szaszi and Darinka Tr{\"u}butschek and Francis Tuerlinckx and Uhlmann, {Eric L.} and Wolf Vanpaemel and Jelte Wicherts and Wagenmakers, {Eric Jan}",

note = "Publisher Copyright: {\textcopyright} 2024 The Authors.",

year = "2024",

month = jul,

day = "24",

doi = "10.1098/rsos.240125",

language = "English (US)",

volume = "11",

journal = "Royal Society Open Science",

issn = "2054-5703",

publisher = "The Royal Society",

number = "7",

}

TY - JOUR

T1 - Subjective evidence evaluation survey for many-analysts studies

AU - Sarafoglou, Alexandra

AU - Hoogeveen, Suzanne

AU - Van Den Bergh, Don

AU - Aczel, Balazs

AU - Albers, Casper J.

AU - Althoff, Tim

AU - Botvinik-Nezer, Rotem

AU - Busch, Niko A.

AU - Cataldo, Andrea M.

AU - Devezer, Berna

AU - Van Dongen, Noah N.N.

AU - Dreber, Anna

AU - Fried, Eiko I.

AU - Hoekstra, Rink

AU - Hoffman, Sabine

AU - Holzmeister, Felix

AU - Huber, Jürgen

AU - Huntington-Klein, Nick

AU - Ioannidis, John

AU - Johannesson, Magnus

AU - Kirchler, Michael

AU - Loken, Eric

AU - Mangin, Jan Francois

AU - Matzke, Dora

AU - Menkveld, Albert J.

AU - Nilsonne, Gustav

AU - Van Ravenzwaaij, Don

AU - Schweinsberg, Martin

AU - Schulz-Kuempel, Hannah

AU - Shanks, David R.

AU - Simons, Daniel J.

AU - Spellman, Barbara A.

AU - Stoevenbelt, Andrea H.

AU - Szaszi, Barnabas

AU - Trübutschek, Darinka

AU - Tuerlinckx, Francis

AU - Uhlmann, Eric L.

AU - Vanpaemel, Wolf

AU - Wicherts, Jelte

AU - Wagenmakers, Eric Jan

PY - 2024/7/24

Y1 - 2024/7/24

N2 - Many-analysts studies explore how well an empirical claim withstands plausible alternative analyses of the same dataset by multiple, independent analysis teams. Conclusions from these studies typically rely on a single outcome metric (e.g. effect size) provided by each analysis team. Although informative about the range of plausible effects in a dataset, a single effect size from each team does not provide a complete, nuanced understanding of how analysis choices are related to the outcome. We used the Delphi consensus technique with input from 37 experts to develop an 18-item subjective evidence evaluation survey (SEES) to evaluate how each analysis team views the methodological appropriateness of the research design and the strength of evidence for the hypothesis. We illustrate the usefulness of the SEES in providing richer evidence assessment with pilot data from a previous many-analysts study.

AB - Many-analysts studies explore how well an empirical claim withstands plausible alternative analyses of the same dataset by multiple, independent analysis teams. Conclusions from these studies typically rely on a single outcome metric (e.g. effect size) provided by each analysis team. Although informative about the range of plausible effects in a dataset, a single effect size from each team does not provide a complete, nuanced understanding of how analysis choices are related to the outcome. We used the Delphi consensus technique with input from 37 experts to develop an 18-item subjective evidence evaluation survey (SEES) to evaluate how each analysis team views the methodological appropriateness of the research design and the strength of evidence for the hypothesis. We illustrate the usefulness of the SEES in providing richer evidence assessment with pilot data from a previous many-analysts study.

KW - crowdsourcing analysis

KW - metascience

KW - open science

KW - scientific transparency

KW - team science

UR - http://www.scopus.com/inward/record.url?scp=85200718634&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85200718634&partnerID=8YFLogxK

U2 - 10.1098/rsos.240125

DO - 10.1098/rsos.240125

M3 - Article

C2 - 39050728

AN - SCOPUS:85200718634

SN - 2054-5703

VL - 11

JO - Royal Society Open Science

JF - Royal Society Open Science

IS - 7

M1 - 240125

ER -

Subjective evidence evaluation survey for many-analysts studies

Abstract

Keywords

ASJC Scopus subject areas

Online availability

Library availability

Related links

Fingerprint

Cite this