EEGtoText: Learning to Write Medical Reports from EEG Recordings

Siddharth Biswal, Cao Xiao, M. Brandon Westover, Jimeng Sun

Research output: Contribution to journalConference articlepeer-review


Electroencephalography (EEG) is widely used in hospitals and clinics for the diagnosis of many neurological conditions. Such diagnoses require accurate and timely clinical reports to summarize the findings from raw EEG data. In this paper, we investigate whether it is possible to automatically generate text reports directly from EEG data. To address the challenges, we proposed EEGtoText, which first extracted shift invariant and temporal patterns using stacked convolutional neural networks and recurrent neural networks (RCNN). These temporal patterns are used to classify key phenotypes including EEG normality, sleep, generalized and focal slowing, epileptiform discharges, spindles, vertex waves and seizures. Based on these phenotypes, the impression section of the EEG report is generated. Next, we adopted a hierarchical long short-term memory network(LSTM) that comprises of paragraph-level and sentence-level LSTMs to generate the detail explanation of the impression. Within the hierarchical LSTM, we used an attention module to localize the abnormal areas in the EEG which provide another explanation and justification of the extracted phenotypes. We conducted large-scale evaluations on two different EEG datasets Dataset1 (n=12,980) and TUH (n=16,950). We achieved an area under the ROC curve (AUC) between.658 to.915 on phenotype classification, which is significantly higher than CRNN and RCNN with attention. We also conducted a quantitative evaluation of the detailed explanation, which achieved METEOR score.371 and BLEU score 4.583. Finally, our initial clinical reviews confirmed the effectiveness of the generated reports.

Original languageEnglish (US)
Pages (from-to)513-531
Number of pages19
JournalProceedings of Machine Learning Research
StatePublished - 2019
Externally publishedYes
Event4th Machine Learning for Healthcare Conference, MLHC 2019 - Ann Arbor, United States
Duration: Aug 9 2019Aug 10 2019

ASJC Scopus subject areas

  • Artificial Intelligence
  • Software
  • Control and Systems Engineering
  • Statistics and Probability


Dive into the research topics of 'EEGtoText: Learning to Write Medical Reports from EEG Recordings'. Together they form a unique fingerprint.

Cite this