Overestimation of action-game training effects: Publication bias and Salami slicing

Joseph Hilgard, Giovanni Sala, Walter R. Boot, Daniel J. Simons

Research output: Contribution to journalArticle

Abstract

Does playing action video games improve performance on tests of cognitive ability? A recent meta-analysis (Bediou et al., 2018a) summarized the available evidence and concluded that it can. Their analysis, however, did not adequately correct for publication bias. We re-analyzed the same set of studies with more appropriate adjustments for publication bias and found minimal evidence for transfer of training to cognitive ability measures. Instead, it is possible that there are little or no benefits, just publication bias — the exclusion of non-significant results from the published literature. That bias may be the cause of a lab effect reported in the original meta-analysis. The meta-analysis showed that studies from the Bavelier lab (the senior author of the meta-analysis) reported larger effects than other labs. We show that many of these original studies distributed different outcomes from the same or highly overlapping sets of participants across publications without noting the overlap. This salami-slicing might contribute to the extent of publication bias in the literature. More compelling, independent, and transparent evidence is needed before concluding that action video game training transfers to performance on other cognitive tasks.

Original languageEnglish (US)
Article number30
JournalCollabra: Psychology
Volume5
Issue number1
DOIs
StatePublished - Jan 1 2019

Fingerprint

Publication Bias
Meta-Analysis
Video Games
Aptitude
Publications
Transfer (Psychology)

Keywords

  • Action game training
  • Cognitive training
  • Meta-analysis
  • Publication bias
  • Salami slicing
  • Video game training

ASJC Scopus subject areas

  • Psychology(all)

Cite this

Overestimation of action-game training effects : Publication bias and Salami slicing. / Hilgard, Joseph; Sala, Giovanni; Boot, Walter R.; Simons, Daniel J.

In: Collabra: Psychology, Vol. 5, No. 1, 30, 01.01.2019.

Research output: Contribution to journalArticle

@article{33672d9cdef54e8ba7739a94b4cb50dc,
title = "Overestimation of action-game training effects: Publication bias and Salami slicing",
abstract = "Does playing action video games improve performance on tests of cognitive ability? A recent meta-analysis (Bediou et al., 2018a) summarized the available evidence and concluded that it can. Their analysis, however, did not adequately correct for publication bias. We re-analyzed the same set of studies with more appropriate adjustments for publication bias and found minimal evidence for transfer of training to cognitive ability measures. Instead, it is possible that there are little or no benefits, just publication bias — the exclusion of non-significant results from the published literature. That bias may be the cause of a lab effect reported in the original meta-analysis. The meta-analysis showed that studies from the Bavelier lab (the senior author of the meta-analysis) reported larger effects than other labs. We show that many of these original studies distributed different outcomes from the same or highly overlapping sets of participants across publications without noting the overlap. This salami-slicing might contribute to the extent of publication bias in the literature. More compelling, independent, and transparent evidence is needed before concluding that action video game training transfers to performance on other cognitive tasks.",
keywords = "Action game training, Cognitive training, Meta-analysis, Publication bias, Salami slicing, Video game training",
author = "Joseph Hilgard and Giovanni Sala and Boot, {Walter R.} and Simons, {Daniel J.}",
year = "2019",
month = "1",
day = "1",
doi = "10.1525/collabra.231",
language = "English (US)",
volume = "5",
journal = "Collabra: Psychology",
issn = "2474-7394",
publisher = "University of California Press",
number = "1",

}

TY - JOUR

T1 - Overestimation of action-game training effects

T2 - Publication bias and Salami slicing

AU - Hilgard, Joseph

AU - Sala, Giovanni

AU - Boot, Walter R.

AU - Simons, Daniel J.

PY - 2019/1/1

Y1 - 2019/1/1

N2 - Does playing action video games improve performance on tests of cognitive ability? A recent meta-analysis (Bediou et al., 2018a) summarized the available evidence and concluded that it can. Their analysis, however, did not adequately correct for publication bias. We re-analyzed the same set of studies with more appropriate adjustments for publication bias and found minimal evidence for transfer of training to cognitive ability measures. Instead, it is possible that there are little or no benefits, just publication bias — the exclusion of non-significant results from the published literature. That bias may be the cause of a lab effect reported in the original meta-analysis. The meta-analysis showed that studies from the Bavelier lab (the senior author of the meta-analysis) reported larger effects than other labs. We show that many of these original studies distributed different outcomes from the same or highly overlapping sets of participants across publications without noting the overlap. This salami-slicing might contribute to the extent of publication bias in the literature. More compelling, independent, and transparent evidence is needed before concluding that action video game training transfers to performance on other cognitive tasks.

AB - Does playing action video games improve performance on tests of cognitive ability? A recent meta-analysis (Bediou et al., 2018a) summarized the available evidence and concluded that it can. Their analysis, however, did not adequately correct for publication bias. We re-analyzed the same set of studies with more appropriate adjustments for publication bias and found minimal evidence for transfer of training to cognitive ability measures. Instead, it is possible that there are little or no benefits, just publication bias — the exclusion of non-significant results from the published literature. That bias may be the cause of a lab effect reported in the original meta-analysis. The meta-analysis showed that studies from the Bavelier lab (the senior author of the meta-analysis) reported larger effects than other labs. We show that many of these original studies distributed different outcomes from the same or highly overlapping sets of participants across publications without noting the overlap. This salami-slicing might contribute to the extent of publication bias in the literature. More compelling, independent, and transparent evidence is needed before concluding that action video game training transfers to performance on other cognitive tasks.

KW - Action game training

KW - Cognitive training

KW - Meta-analysis

KW - Publication bias

KW - Salami slicing

KW - Video game training

UR - http://www.scopus.com/inward/record.url?scp=85072024210&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85072024210&partnerID=8YFLogxK

U2 - 10.1525/collabra.231

DO - 10.1525/collabra.231

M3 - Article

AN - SCOPUS:85072024210

VL - 5

JO - Collabra: Psychology

JF - Collabra: Psychology

SN - 2474-7394

IS - 1

M1 - 30

ER -