Bootstrap-Based Between-Study Heterogeneity Tests in Meta-Analysis

Han Du, Ge Jiang, Zijun Ke

Research output: Contribution to journalArticlepeer-review


Meta-analysis combines pertinent information from existing studies to provide an overall estimate of population parameters/effect sizes, as well as to quantify and explain the differences between studies. However, testing between-study heterogeneity is one of the most challenging tasks in meta-analysis research. Existing methods for testing heterogeneity, such as the Q test and likelihood ratio (LR) test, have been criticized for their failure to control Type I error rate and/or failure to attain enough statistical power. Although better reference distribution approximations have been proposed in the literature, their application is limited. Additionally, when the interest is to test whether the size of the heterogeneity is larger than a specific level, existing methods are far from mature. To address these issues, we propose new heterogeneity tests. Specifically, we combine bootstrap methods with existing heterogeneity tests (i.e., the maximum LR test, the restricted maximum LR test, and the Q test) to overcome the reference distribution issue and denote them as B-ML-LRT, B-REML-LRT, and B-Q, respectively. Simulation studies were conducted to examine and compare the performance of the proposed methods with the regular LR test, the regular Q test, and the Kulinskaya’s improved Q test in both random- and mixed-effects meta-analyses. Based on the results of Type I error rates and statistical power, B-REML-LRT is recommended. Additionally, the improved Q test is also recommended when it is applicable. An R package (Formula presented.) is provided to facilitate the implementation of the proposed tests.

Original languageEnglish (US)
Pages (from-to)484-503
Number of pages20
JournalMultivariate Behavioral Research
Issue number3
StatePublished - 2023


  • Meta-analysis
  • between-study heterogeneity
  • bootstrap method

ASJC Scopus subject areas

  • Experimental and Cognitive Psychology
  • Arts and Humanities (miscellaneous)
  • Statistics and Probability


Dive into the research topics of 'Bootstrap-Based Between-Study Heterogeneity Tests in Meta-Analysis'. Together they form a unique fingerprint.

Cite this