Query polyrepresentation for ranking retrieval systems without relevance judgments

Miles Efron, Megan Winget

Research output: Contribution to journalArticlepeer-review

Abstract

Ranking information retrieval (IR) systems with respect to their effectiveness is a crucial operation during IR evaluation, as well as during data fusion. This article offers a novel method of approaching the system-ranking problem, based on the widely studied idea of polyrepresentation. The principle of polyrepresentation suggests that a single information need can be represented by many query articulations-what we call query aspects. By skimming the top k (where k is small) documents retrieved by a single system for multiple query aspects, we collect a set of documents that are likely to be relevant to a given test topic. Labeling these skimmed documents as putatively relevant lets us build pseudorelevance judgments without undue human intervention. We report experiments where using these pseudorelevance judgments delivers a rank ordering of IR systems that correlates highly with rankings based on human relevance judgments.

Original languageEnglish (US)
Pages (from-to)1081-1091
Number of pages11
JournalJournal of the American Society for Information Science and Technology
Volume61
Issue number6
DOIs
StatePublished - Jun 1 2010

ASJC Scopus subject areas

  • Software
  • Information Systems
  • Human-Computer Interaction
  • Computer Networks and Communications
  • Artificial Intelligence

Fingerprint Dive into the research topics of 'Query polyrepresentation for ranking retrieval systems without relevance judgments'. Together they form a unique fingerprint.

Cite this