TY - JOUR
T1 - Subjective evidence evaluation survey for many-analysts studies
AU - Sarafoglou, Alexandra
AU - Hoogeveen, Suzanne
AU - Van Den Bergh, Don
AU - Aczel, Balazs
AU - Albers, Casper J.
AU - Althoff, Tim
AU - Botvinik-Nezer, Rotem
AU - Busch, Niko A.
AU - Cataldo, Andrea M.
AU - Devezer, Berna
AU - Van Dongen, Noah N.N.
AU - Dreber, Anna
AU - Fried, Eiko I.
AU - Hoekstra, Rink
AU - Hoffman, Sabine
AU - Holzmeister, Felix
AU - Huber, Jürgen
AU - Huntington-Klein, Nick
AU - Ioannidis, John
AU - Johannesson, Magnus
AU - Kirchler, Michael
AU - Loken, Eric
AU - Mangin, Jan Francois
AU - Matzke, Dora
AU - Menkveld, Albert J.
AU - Nilsonne, Gustav
AU - Van Ravenzwaaij, Don
AU - Schweinsberg, Martin
AU - Schulz-Kuempel, Hannah
AU - Shanks, David R.
AU - Simons, Daniel J.
AU - Spellman, Barbara A.
AU - Stoevenbelt, Andrea H.
AU - Szaszi, Barnabas
AU - Trübutschek, Darinka
AU - Tuerlinckx, Francis
AU - Uhlmann, Eric L.
AU - Vanpaemel, Wolf
AU - Wicherts, Jelte
AU - Wagenmakers, Eric Jan
N1 - We thank Fr\u00E9d\u00E9rique Heppel for her support in creating the initial item list of the SEES and helping with data collection. This work was supported by an Amsterdam Brain and Cognition (ABC) project grant to A.S. (ABC PG 22). T.A. was supported in part by NSF IIS-1901386, NSF CAREER IIS-2142794. N.A.B. is supported by the German Research Foundation (DFG; BU 2400/11-1) and by the DFG priority programme \u2018META-REP: A Meta-scientific Programme to Analyse and Optimize Replicability in the Behavioural, Social, and Cognitive Sciences\u2019. B.D. was supported by the National Institute of General Medical Sciences of the National Institutes of Health under Award Number P20GM104420. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. N.N.N.v.D. was supported by the NWO Vici grant no. 181.029. A.D. was supported by the Knut and Alice Wallenberg Foundation, the Marianne and Marcus Wallenberg Foundation and the Jan Wallander and Tom Hedelius Foundation. D.M. is supported by a Vidi grant (VI.Vidi.191.091) from the Dutch Research Council (NWO). A.J.M. is supported by a Vici grant (016.Vici.185.068) from the Dutch Research Council (NWO). D.v.R. is supported by a Vidi grant (016.Vidi.188.001) from the Dutch Research Council (NWO). D.R.S. was supported by a grant from the UK Economic and Social Research Council. D.T. was supported by Marie Sklodowska-Curie grant agreement no. 101023805 under the European Union\u2019s Horizon 2020 research and innovation programme. E.L.U. was supported by an R&D grant from INSEAD. J.W. is supported by a VICI grant (VI.C.221.100) from the Dutch Research Council (NWO).
N.N.N.v.D. was supported by the NWO Vici grant no. 181.029. A.D. was supported by the Knut and Alice Wallenberg Foundation, the Marianne and Marcus Wallenberg Foundation and the Jan Wallander and Tom Hedelius Foundation. D.M. is supported by a Vidi grant (VI.Vidi.191.091) from the Dutch Research Council (NWO). A.J.M. is supported by a Vici grant (016.Vici.185.068) from the Dutch Research Council (NWO). D.v.R. is supported by a Vidi grant (016.Vidi.188.001) from the Dutch Research Council (NWO). D.R.S. was supported by a grant from the UK Economic and Social Research Council. D.T. was supported by Marie Sklodowska-Curie grant agreement no. 101023805 under the European Union s Horizon 2020 research and innovation programme. E.L.U. was supported by an R&D grant from INSEAD. J.W. is supported by a VICI grant (VI.C.221.100) from the Dutch Research Council (NWO).
PY - 2024/7/24
Y1 - 2024/7/24
N2 - Many-analysts studies explore how well an empirical claim withstands plausible alternative analyses of the same dataset by multiple, independent analysis teams. Conclusions from these studies typically rely on a single outcome metric (e.g. effect size) provided by each analysis team. Although informative about the range of plausible effects in a dataset, a single effect size from each team does not provide a complete, nuanced understanding of how analysis choices are related to the outcome. We used the Delphi consensus technique with input from 37 experts to develop an 18-item subjective evidence evaluation survey (SEES) to evaluate how each analysis team views the methodological appropriateness of the research design and the strength of evidence for the hypothesis. We illustrate the usefulness of the SEES in providing richer evidence assessment with pilot data from a previous many-analysts study.
AB - Many-analysts studies explore how well an empirical claim withstands plausible alternative analyses of the same dataset by multiple, independent analysis teams. Conclusions from these studies typically rely on a single outcome metric (e.g. effect size) provided by each analysis team. Although informative about the range of plausible effects in a dataset, a single effect size from each team does not provide a complete, nuanced understanding of how analysis choices are related to the outcome. We used the Delphi consensus technique with input from 37 experts to develop an 18-item subjective evidence evaluation survey (SEES) to evaluate how each analysis team views the methodological appropriateness of the research design and the strength of evidence for the hypothesis. We illustrate the usefulness of the SEES in providing richer evidence assessment with pilot data from a previous many-analysts study.
KW - crowdsourcing analysis
KW - metascience
KW - open science
KW - scientific transparency
KW - team science
UR - http://www.scopus.com/inward/record.url?scp=85200718634&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85200718634&partnerID=8YFLogxK
U2 - 10.1098/rsos.240125
DO - 10.1098/rsos.240125
M3 - Article
C2 - 39050728
AN - SCOPUS:85200718634
SN - 2054-5703
VL - 11
JO - Royal Society Open Science
JF - Royal Society Open Science
IS - 7
M1 - 240125
ER -