Confirm or refute?

A comparative study on citation sentiment classification in clinical research publications

Halil Kilicoglu, Zeshan Peng, Shabnam Tafreshi, Tung Tran, Graciela Rosemblat, Jodi A Schneider

Research output: Contribution to journalArticle

Abstract

Quantifying scientific impact of researchers and journals relies largely on citation counts, despite the acknowledged limitations of this approach. The need for more suitable alternatives has prompted research into developing advanced metrics, such as h-index and Relative Citation Ratio (RCR), as well as better citation categorization schemes to capture the various functions that citations serve in a publication. One such scheme involves citation sentiment: whether a reference paper is cited positively (agreement with the findings of the reference paper), negatively (disagreement), or neutrally. The ability to classify citation function in this manner can be viewed as a first step toward a more fine-grained bibliometrics. In this study, we compared several approaches, varying in complexity, for classification of citation sentiment in clinical trial publications. Using a corpus of 285 discussion sections from as many publications (a total of 4,182 citations), we developed a rule-based method as well as supervised machine learning models based on support vector machines (SVM) and two variants of deep neural networks; namely, convolutional neural network (CNN) and bidirectional long short-term memory (BiLSTM). A CNN model augmented with hand-crafted features yielded the best performance (0.882 accuracy and 0.721 macro-F 1 on held-out set). Our results show that baseline performances of traditional supervised learning algorithms and deep neural network architectures are similar and that hand-crafted features based on sentiment dictionaries and rhetorical structure allow neural network approaches to outperform traditional machine learning approaches for this task. We make the rule-based method and the best-performing neural network model publicly available at: https://github.com/kilicogluh/clinical-citation-sentiment.

Original languageEnglish (US)
Article number103123
JournalJournal of Biomedical Informatics
Volume91
DOIs
StatePublished - Mar 1 2019

Fingerprint

Publications
Neural Networks (Computer)
Neural networks
Hand
Research
Bibliometrics
Learning systems
Aptitude
Long-Term Memory
Short-Term Memory
Supervised learning
Glossaries
Network architecture
Research Personnel
Clinical Trials
Learning
Learning algorithms
Support vector machines
Macros
Deep neural networks

Keywords

  • Citation analysis
  • Natural language processing
  • Neural networks
  • Sentiment analysis
  • Supervised machine learning

ASJC Scopus subject areas

  • Computer Science Applications
  • Health Informatics

Cite this

Confirm or refute? A comparative study on citation sentiment classification in clinical research publications. / Kilicoglu, Halil; Peng, Zeshan; Tafreshi, Shabnam; Tran, Tung; Rosemblat, Graciela; Schneider, Jodi A.

In: Journal of Biomedical Informatics, Vol. 91, 103123, 01.03.2019.

Research output: Contribution to journalArticle

Kilicoglu, Halil ; Peng, Zeshan ; Tafreshi, Shabnam ; Tran, Tung ; Rosemblat, Graciela ; Schneider, Jodi A. / Confirm or refute? A comparative study on citation sentiment classification in clinical research publications. In: Journal of Biomedical Informatics. 2019 ; Vol. 91.
@article{2f30864c1440405c9b0b4d798c95c782,
title = "Confirm or refute?: A comparative study on citation sentiment classification in clinical research publications",
abstract = "Quantifying scientific impact of researchers and journals relies largely on citation counts, despite the acknowledged limitations of this approach. The need for more suitable alternatives has prompted research into developing advanced metrics, such as h-index and Relative Citation Ratio (RCR), as well as better citation categorization schemes to capture the various functions that citations serve in a publication. One such scheme involves citation sentiment: whether a reference paper is cited positively (agreement with the findings of the reference paper), negatively (disagreement), or neutrally. The ability to classify citation function in this manner can be viewed as a first step toward a more fine-grained bibliometrics. In this study, we compared several approaches, varying in complexity, for classification of citation sentiment in clinical trial publications. Using a corpus of 285 discussion sections from as many publications (a total of 4,182 citations), we developed a rule-based method as well as supervised machine learning models based on support vector machines (SVM) and two variants of deep neural networks; namely, convolutional neural network (CNN) and bidirectional long short-term memory (BiLSTM). A CNN model augmented with hand-crafted features yielded the best performance (0.882 accuracy and 0.721 macro-F 1 on held-out set). Our results show that baseline performances of traditional supervised learning algorithms and deep neural network architectures are similar and that hand-crafted features based on sentiment dictionaries and rhetorical structure allow neural network approaches to outperform traditional machine learning approaches for this task. We make the rule-based method and the best-performing neural network model publicly available at: https://github.com/kilicogluh/clinical-citation-sentiment.",
keywords = "Citation analysis, Natural language processing, Neural networks, Sentiment analysis, Supervised machine learning",
author = "Halil Kilicoglu and Zeshan Peng and Shabnam Tafreshi and Tung Tran and Graciela Rosemblat and Schneider, {Jodi A}",
year = "2019",
month = "3",
day = "1",
doi = "10.1016/j.jbi.2019.103123",
language = "English (US)",
volume = "91",
journal = "Journal of Biomedical Informatics",
issn = "1532-0464",
publisher = "Academic Press Inc.",

}

TY - JOUR

T1 - Confirm or refute?

T2 - A comparative study on citation sentiment classification in clinical research publications

AU - Kilicoglu, Halil

AU - Peng, Zeshan

AU - Tafreshi, Shabnam

AU - Tran, Tung

AU - Rosemblat, Graciela

AU - Schneider, Jodi A

PY - 2019/3/1

Y1 - 2019/3/1

N2 - Quantifying scientific impact of researchers and journals relies largely on citation counts, despite the acknowledged limitations of this approach. The need for more suitable alternatives has prompted research into developing advanced metrics, such as h-index and Relative Citation Ratio (RCR), as well as better citation categorization schemes to capture the various functions that citations serve in a publication. One such scheme involves citation sentiment: whether a reference paper is cited positively (agreement with the findings of the reference paper), negatively (disagreement), or neutrally. The ability to classify citation function in this manner can be viewed as a first step toward a more fine-grained bibliometrics. In this study, we compared several approaches, varying in complexity, for classification of citation sentiment in clinical trial publications. Using a corpus of 285 discussion sections from as many publications (a total of 4,182 citations), we developed a rule-based method as well as supervised machine learning models based on support vector machines (SVM) and two variants of deep neural networks; namely, convolutional neural network (CNN) and bidirectional long short-term memory (BiLSTM). A CNN model augmented with hand-crafted features yielded the best performance (0.882 accuracy and 0.721 macro-F 1 on held-out set). Our results show that baseline performances of traditional supervised learning algorithms and deep neural network architectures are similar and that hand-crafted features based on sentiment dictionaries and rhetorical structure allow neural network approaches to outperform traditional machine learning approaches for this task. We make the rule-based method and the best-performing neural network model publicly available at: https://github.com/kilicogluh/clinical-citation-sentiment.

AB - Quantifying scientific impact of researchers and journals relies largely on citation counts, despite the acknowledged limitations of this approach. The need for more suitable alternatives has prompted research into developing advanced metrics, such as h-index and Relative Citation Ratio (RCR), as well as better citation categorization schemes to capture the various functions that citations serve in a publication. One such scheme involves citation sentiment: whether a reference paper is cited positively (agreement with the findings of the reference paper), negatively (disagreement), or neutrally. The ability to classify citation function in this manner can be viewed as a first step toward a more fine-grained bibliometrics. In this study, we compared several approaches, varying in complexity, for classification of citation sentiment in clinical trial publications. Using a corpus of 285 discussion sections from as many publications (a total of 4,182 citations), we developed a rule-based method as well as supervised machine learning models based on support vector machines (SVM) and two variants of deep neural networks; namely, convolutional neural network (CNN) and bidirectional long short-term memory (BiLSTM). A CNN model augmented with hand-crafted features yielded the best performance (0.882 accuracy and 0.721 macro-F 1 on held-out set). Our results show that baseline performances of traditional supervised learning algorithms and deep neural network architectures are similar and that hand-crafted features based on sentiment dictionaries and rhetorical structure allow neural network approaches to outperform traditional machine learning approaches for this task. We make the rule-based method and the best-performing neural network model publicly available at: https://github.com/kilicogluh/clinical-citation-sentiment.

KW - Citation analysis

KW - Natural language processing

KW - Neural networks

KW - Sentiment analysis

KW - Supervised machine learning

UR - http://www.scopus.com/inward/record.url?scp=85061587467&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85061587467&partnerID=8YFLogxK

U2 - 10.1016/j.jbi.2019.103123

DO - 10.1016/j.jbi.2019.103123

M3 - Article

VL - 91

JO - Journal of Biomedical Informatics

JF - Journal of Biomedical Informatics

SN - 1532-0464

M1 - 103123

ER -