Diagnostic evaluation of information retrieval models

Hui Fang, Tao Tao, Chengxiang Zhai

Research output: Contribution to journalArticlepeer-review


Developing effective retrieval models is a long-standing central challenge in information retrieval research. In order to develop more effective models, it is necessary to understand the deficiencies of the current retrieval models and the relative strengths of each of them. In this article, we propose a general methodology to analytically and experimentally diagnose the weaknesses of a retrieval function, which provides guidance on how to further improve its performance. Our methodology is motivated by the empirical observation that good retrieval performance is closely related to the use of various retrieval heuristics. We connect the weaknesses and strengths of a retrieval function with its implementations of these retrieval heuristics, and propose two strategies to check how well a retrieval function implements the desired retrieval heuristics. The first strategy is to formalize heuristics as constraints, and use constraint analysis to analytically check the implementation of retrieval heuristics. The second strategy is to define a set of relevance-preserving perturbations and perform diagnostic tests to empirically evaluate how well a retrieval function implements retrieval heuristics. Experiments show that both strategies are effective to identify the potential problems in implementations of the retrieval heuristics. The performance of retrieval functions can be improved after we fix these problems.

Original languageEnglish (US)
Article number7
JournalACM Transactions on Information Systems
Issue number2
StatePublished - Apr 2011


  • Constraints
  • Diagnostic evaluation
  • Formal models
  • Retrieval heuristics
  • TF-IDF weighting

ASJC Scopus subject areas

  • Information Systems
  • Business, Management and Accounting(all)
  • Computer Science Applications


Dive into the research topics of 'Diagnostic evaluation of information retrieval models'. Together they form a unique fingerprint.

Cite this