TILM: A neural language model with evolving topical influence

Shubhra Kanti Karmaker Santu, Kalyan Veeramachaneni, Cheng Xiang Zhai

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Content of text data are often influenced by contextual factors which often evolve over time (e.g., content of social media are often influenced by topics covered in the major news streams). Existing language models do not consider the influence of such related evolving topics, and thus are not optimal. In this paper, we propose to incorporate such topical-influence into a language model to both improve its accuracy and enable cross-stream analysis of topical influences. Specifically, we propose a novel language model called Topical Influence Language Model (TILM), which is a novel extension of a neural language model to capture the influences on the contents in one text stream by the evolving topics in another related (or possibly same) text stream. Experimental results on six different text stream data comprised of conference paper titles show that the incorporation of evolving topical influence into a language model is beneficial and TILM outperforms multiple baselines in a challenging task of text forecasting. In addition to serving as a language model, TILM further enables interesting analysis of topical influence among multiple text streams.

Original languageEnglish (US)
Title of host publicationCoNLL 2019 - 23rd Conference on Computational Natural Language Learning, Proceedings of the Conference
PublisherAssociation for Computational Linguistics
Pages778-788
Number of pages11
ISBN (Electronic)9781950737727
StatePublished - 2019
Event23rd Conference on Computational Natural Language Learning, CoNLL 2019 - Hong Kong, China
Duration: Nov 3 2019Nov 4 2019

Publication series

NameCoNLL 2019 - 23rd Conference on Computational Natural Language Learning, Proceedings of the Conference

Conference

Conference23rd Conference on Computational Natural Language Learning, CoNLL 2019
CountryChina
CityHong Kong
Period11/3/1911/4/19

ASJC Scopus subject areas

  • Computer Science Applications
  • Information Systems
  • Computational Theory and Mathematics

Cite this