TY - JOUR
T1 - Evaluation of publication type tagging as a strategy to screen randomized controlled trial articles in preparing systematic reviews
AU - Schneider, Jodi
AU - Hoang, Linh
AU - Kansara, Yogeshwar
AU - Cohen, Aaron M
AU - Smalheiser, Neil R
N1 - Funding Information:
This study was funded by a grant from the National Library of Medicine, "Text Mining Pipeline to Accelerate Systematic Reviews in Evidence-based Medicine" (R01LM010817). The funding agency had no role in the preparation, review, or approval of the manuscript. The opinions, results, and conclusions reported in this paper are those of the authors and are independent of the funding source.
Publisher Copyright:
© 2022 The Author(s).
PY - 2022/4/1
Y1 - 2022/4/1
N2 - Objectives: To produce a systematic review (SR), reviewers typically screen thousands of titles and abstracts of articles manually to find a small number which are read in full text to find relevant articles included in the final SR. Here, we evaluate a proposed automated probabilistic publication type screening strategy applied to the randomized controlled trial (RCT) articles (i.e., those which present clinical outcome results of RCT studies) included in a corpus of previously published Cochrane reviews. Materials and Methods: We selected a random subset of 558 published Cochrane reviews that specified RCT study only inclusion criteria, containing 7113 included articles which could be matched to PubMed identifiers. These were processed by our automated RCT Tagger tool to estimate the probability that each article reports clinical outcomes of a RCT. Results: Removing articles with low predictive scores P < 0.01 eliminated 288 included articles, of which only 22 were actually typical RCT articles, and only 18 were actually typical RCT articles that MEDLINE indexed as such. Based on our sample set, this screening strategy led to fewer than 0.05 relevant RCT articles being missed on average per Cochrane SR. Discussion: This scenario, based on real SRs, demonstrates that automated tagging can identify RCT articles accurately while maintaining very high recall. However, we also found that even SRs whose inclusion criteria are restricted to RCT studies include not only clinical outcome articles per se, but a variety of ancillary article types as well. Conclusions: This encourages further studies learning how best to incorporate automated tagging of additional publication types into SR triage workflows.
AB - Objectives: To produce a systematic review (SR), reviewers typically screen thousands of titles and abstracts of articles manually to find a small number which are read in full text to find relevant articles included in the final SR. Here, we evaluate a proposed automated probabilistic publication type screening strategy applied to the randomized controlled trial (RCT) articles (i.e., those which present clinical outcome results of RCT studies) included in a corpus of previously published Cochrane reviews. Materials and Methods: We selected a random subset of 558 published Cochrane reviews that specified RCT study only inclusion criteria, containing 7113 included articles which could be matched to PubMed identifiers. These were processed by our automated RCT Tagger tool to estimate the probability that each article reports clinical outcomes of a RCT. Results: Removing articles with low predictive scores P < 0.01 eliminated 288 included articles, of which only 22 were actually typical RCT articles, and only 18 were actually typical RCT articles that MEDLINE indexed as such. Based on our sample set, this screening strategy led to fewer than 0.05 relevant RCT articles being missed on average per Cochrane SR. Discussion: This scenario, based on real SRs, demonstrates that automated tagging can identify RCT articles accurately while maintaining very high recall. However, we also found that even SRs whose inclusion criteria are restricted to RCT studies include not only clinical outcome articles per se, but a variety of ancillary article types as well. Conclusions: This encourages further studies learning how best to incorporate automated tagging of additional publication types into SR triage workflows.
KW - information retrieval
KW - randomized controlled trials
KW - RCT Tagger
KW - systematic review automation
UR - http://www.scopus.com/inward/record.url?scp=85142083703&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85142083703&partnerID=8YFLogxK
U2 - 10.1093/jamiaopen/ooac015
DO - 10.1093/jamiaopen/ooac015
M3 - Article
C2 - 35571360
SN - 2574-2531
VL - 5
JO - JAMIA Open
JF - JAMIA Open
IS - 1
M1 - ooac015
ER -