Abstract

Although systematic reviews are intended to provide trusted scientific knowledge to meet the needs of decision-makers, their reliability can be threatened by bias and irreproducibility. To help decision-makers assess the risks in systematic reviews that they intend to use as the foundation of their action, we designed and tested a new approach to analyzing the evidence selection of a review: its coverage of the primary literature and its comparison to other reviews. Our approach could also help anyone using or producing reviews understand diversity or convergence in evidence selection. The basis of our approach is a new network construct called the inclusion network, which has two types of nodes: primary study reports (PSRs, the evidence) and systematic review reports (SRRs). The approach assesses risks in a given systematic review (the target SRR) by first constructing an inclusion network of the target SRR and other systematic reviews studying similar research questions (the companion SRRs) and then applying a three-step assessment process that utilizes visualizations, quantitative network metrics, and time series analysis. This paper introduces our approach and demonstrates it in two case studies. We identified the following risks: missing potentially relevant evidence, epistemic division in the scientific community, and recent instability in evidence selection standards. We also compare our inclusion network approach to knowledge assessment approaches based on another influential network construct, the claim-specific citation network, discuss current limitations of the inclusion network approach, and present directions for future work.

Original languageEnglish (US)
Pages (from-to)219-245
Number of pages27
JournalQuantitative Science Studies
Volume5
Issue number1
DOIs
StatePublished - Mar 1 2024

Keywords

  • claim-specific citation networks
  • decision making
  • evidence selection discordance
  • evidence synthesis
  • Jaccard similarity
  • specialized-purpose citation networks

ASJC Scopus subject areas

  • Analysis
  • Numerical Analysis
  • Cultural Studies
  • Library and Information Sciences

Fingerprint

Dive into the research topics of 'Exploring evidence selection with the inclusion network'. Together they form a unique fingerprint.

Cite this