Information retrieval evaluation as search simulation: A general formal framework for IR evaluation

Yinan Zhang, Xueqing Liu, Chengxiang Zhai

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

While the Cranfield evaluation methodology based on test collections has been very useful for evaluating simple IR systems that return a ranked list of documents, it has significant limitations when applied to search systems with interface features going beyond a ranked list, and sophisticated interactive IR systems in general. In this paper, we propose a general formal framework for evaluating IR systems based on search session simulation that can be used to perform reproducible experiments for evaluating any IR system, including interactive systems and systems with sophisticated interfaces. We show that the traditional Cranfield evaluation method can be regarded as a special instantiation of the proposed framework where the simulated search session is a user sequentially browsing the presented search results. By examining a number of existing evaluation metrics in the proposed framework, we reveal the exact assumptions they have made implicitly about the simulated users and discuss possible ways to improve these metrics. We further show that the proposed framework enables us to evaluate a set of tag-based search interfaces, a generalization of faceted browsing interfaces, producing results consistent with real user experiments and revealing interesting findings about effectiveness of the interfaces for different types of users.

Original languageEnglish (US)
Title of host publicationICTIR 2017 - Proceedings of the 2017 ACM SIGIR International Conference on the Theory of Information Retrieval
PublisherAssociation for Computing Machinery, Inc
Pages193-200
Number of pages8
ISBN (Electronic)9781450344906
DOIs
StatePublished - Oct 1 2017
Event7th ACM SIGIR International Conference on the Theory of Information Retrieval, ICTIR 2017 - Amsterdam, Netherlands
Duration: Oct 1 2017Oct 4 2017

Publication series

NameICTIR 2017 - Proceedings of the 2017 ACM SIGIR International Conference on the Theory of Information Retrieval

Other

Other7th ACM SIGIR International Conference on the Theory of Information Retrieval, ICTIR 2017
CountryNetherlands
CityAmsterdam
Period10/1/1710/4/17

Keywords

  • IR evaluation
  • Interface card
  • User simulation

ASJC Scopus subject areas

  • Information Systems
  • Computer Science (miscellaneous)

Fingerprint Dive into the research topics of 'Information retrieval evaluation as search simulation: A general formal framework for IR evaluation'. Together they form a unique fingerprint.

Cite this