Estimation of statistical translation models based on mutual information for ad hoc information retrieval

Maryam Karimzadehgan, Cheng Xiang Zhai

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

As a principled approach to capturing semantic relations of words in information retrieval, statistical translation models have been shown to outperform simple document language models which rely on exact matching of words in the query and documents. A main challenge in applying translation models to ad hoc information retrieval is to estimate a translation model without training data. Existing work has relied on training on synthetic queries generated based on a document collection. However, this method is computationally expensive and does not have a good coverage of query words. In this paper, we propose an alternative way to estimate a translation model based on normalized mutual information between words, which is less computationally expensive and has better coverage of query words than the synthetic query method of estimation. We also propose to regularize estimated translation probabilities to ensure sufficient probability mass for self-translation. Experiment results show that the proposed mutual information-based estimation method is not only more efficient, but also more effective than the synthetic query-based method, and it can be combined with pseudo-relevance feedback to further improve retrieval accuracy. The results also show that the proposed regularization strategy is effective and can improve retrieval accuracy for both synthetic query-based estimation and mutual information-based estimation.

Original languageEnglish (US)
Title of host publicationSIGIR 2010 Proceedings - 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval
Pages323-330
Number of pages8
DOIs
StatePublished - 2010
Event33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2010 - Geneva, Switzerland
Duration: Jul 19 2010Jul 23 2010

Publication series

NameSIGIR 2010 Proceedings - 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

Other

Other33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2010
CountrySwitzerland
CityGeneva
Period7/19/107/23/10

Keywords

  • Estimation
  • Feedback
  • Language models
  • Smoothing
  • Statistical machine translation

ASJC Scopus subject areas

  • Information Systems

Fingerprint Dive into the research topics of 'Estimation of statistical translation models based on mutual information for ad hoc information retrieval'. Together they form a unique fingerprint.

Cite this