A study of smoothing methods for language models applied to information retrieval

Chengxiang Zhai, John Lafferty

Research output: Contribution to journalArticlepeer-review


Language modeling approaches to information retrieval are attractive and promising because they connect the problem of retrieval with that of language model estimation, which has been studied extensively in other application areas such as speech recognition. The basic idea of these approaches is to estimate a language model for each document, and to then rank documents by the likelihood of the query according to the estimated language model. A central issue in language model estimation is smoothing, the problem of adjusting the maximum likelihood estimator to compensate for data sparseness. In this article, we study the problem of language model smoothing and its influence on retrieval performance. We examine the sensitivity of retrieval performance to the smoothing parameters and compare several popular smoothing methods on different test collections. Experimental results show that not only is the retrieval performance generally sensitive to the smoothing parameters, but also the sensitivity pattern is affected by the query type, with performance being more sensitive to smoothing for verbose queries than for keyword queries. Verbose queries also generally require more aggressive smoothing to achieve optimal performance. This suggests that smoothing plays two different role - to make the estimated document language model more accurate and to "explain" the noninformative words in the query. In order to decouple these two distinct roles of smoothing, we propose a two-stage smoothing strategy, which yields better sensitivity patterns and facilitates the setting of smoothing parameters automatically. We further propose methods for estimating the smoothing parameters automatically. Evaluation on five different databases and four types of queries indicates that the two-stage smoothing method with the proposed parameter estimation methods consistently gives retrieval performance that is close to - or better than - the best results achieved using a single smoothing method and exhaustive parameter search on the test data.

Original languageEnglish (US)
Pages (from-to)179-214
Number of pages36
JournalACM Transactions on Information Systems
Issue number2
StatePublished - Apr 2004


  • Absolute discounting smoothing
  • Backoff smoothing
  • Dirichlet prior smoothing
  • EM algorithm
  • Interpolation smoothing
  • Jelinek-Mercer smoothing
  • Leave-one-out
  • Risk minimization
  • Statistical language models
  • TF-IDF weighting
  • Term weighting
  • Two-stage smoothing

ASJC Scopus subject areas

  • Information Systems
  • General Business, Management and Accounting
  • Computer Science Applications


Dive into the research topics of 'A study of smoothing methods for language models applied to information retrieval'. Together they form a unique fingerprint.

Cite this