Evaluation of publication type tagging as a strategy to screen randomized controlled trial articles in preparing systematic reviews

Jodi Schneider, Linh Hoang, Yogeshwar Kansara, Aaron M Cohen, Neil R Smalheiser

Research output: Contribution to journalArticlepeer-review


Objectives: To produce a systematic review (SR), reviewers typically screen thousands of titles and abstracts of articles manually to find a small number which are read in full text to find relevant articles included in the final SR. Here, we evaluate a proposed automated probabilistic publication type screening strategy applied to the randomized controlled trial (RCT) articles (i.e., those which present clinical outcome results of RCT studies) included in a corpus of previously published Cochrane reviews. Materials and Methods: We selected a random subset of 558 published Cochrane reviews that specified RCT study only inclusion criteria, containing 7113 included articles which could be matched to PubMed identifiers. These were processed by our automated RCT Tagger tool to estimate the probability that each article reports clinical outcomes of a RCT. Results: Removing articles with low predictive scores P < 0.01 eliminated 288 included articles, of which only 22 were actually typical RCT articles, and only 18 were actually typical RCT articles that MEDLINE indexed as such. Based on our sample set, this screening strategy led to fewer than 0.05 relevant RCT articles being missed on average per Cochrane SR. Discussion: This scenario, based on real SRs, demonstrates that automated tagging can identify RCT articles accurately while maintaining very high recall. However, we also found that even SRs whose inclusion criteria are restricted to RCT studies include not only clinical outcome articles per se, but a variety of ancillary article types as well. Conclusions: This encourages further studies learning how best to incorporate automated tagging of additional publication types into SR triage workflows.

Original languageEnglish (US)
Article numberooac015
JournalJAMIA Open
Issue number1
StatePublished - Apr 1 2022


  • information retrieval
  • randomized controlled trials
  • RCT Tagger
  • systematic review automation

ASJC Scopus subject areas

  • Health Informatics


Dive into the research topics of 'Evaluation of publication type tagging as a strategy to screen randomized controlled trial articles in preparing systematic reviews'. Together they form a unique fingerprint.

Cite this