EEGtoText: Learning to Write Medical Reports from EEG Recordings

Electroencephalography (EEG) is widely used in hospitals and clinics for the diagnosis of many neurological conditions. Such diagnoses require accurate and timely clinical reports to summarize the findings from raw EEG data. In this paper, we investigate whether it is possible to automatically generate text reports directly from EEG data. To address the challenges, we proposed EEGtoText, which first extracted shift invariant and temporal patterns using stacked convolutional neural networks and recurrent neural networks (RCNN). These temporal patterns are used to classify key phenotypes including EEG normality, sleep, generalized and focal slowing, epileptiform discharges, spindles, vertex waves and seizures. Based on these phenotypes, the impression section of the EEG report is generated. Next, we adopted a hierarchical long short-term memory network(LSTM) that comprises of paragraph-level and sentence-level LSTMs to generate the detail explanation of the impression. Within the hierarchical LSTM, we used an attention module to localize the abnormal areas in the EEG which provide another explanation and justification of the extracted phenotypes. We conducted large-scale evaluations on two different EEG datasets Dataset1 (n=12,980) and TUH (n=16,950). We achieved an area under the ROC curve (AUC) between.658 to.915 on phenotype classification, which is significantly higher than CRNN and RCNN with attention. We also conducted a quantitative evaluation of the detailed explanation, which achieved METEOR score.371 and BLEU score 4.583. Finally, our initial clinical reviews confirmed the effectiveness of the generated reports.