Using multiple uery aspects to build test collections without human relevance udgments

Miles Efron

doi:10.1007/978-3-642-00958-7_28

Using multiple uery aspects to build test collections without human relevance udgments

Miles Efron

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Collecting relevance judgments (qrels) is an especially challenging part of building an information retrieval test collection. This paper presents a novel method for creating test collections by offering a substitute for relevance judgments. Our method is based on an old idea in IR: a single information need can be represented by many query articulations. We call different articulations of a particular need query aspects. By combining the top k documents retrieved by a single system for multiple query aspects, we build judgment-free qrels whose rank ordering of IR systems correlates highly with rankings based on human relevance judgments.

Original language	English (US)
Title of host publication	Advances in Information Retrieval - 31th European Conference on IR Research, ECIR 2009, Proceedings
Pages	301-312
Number of pages	12
DOIs	https://doi.org/10.1007/978-3-642-00958-7_28
State	Published - 2009
Externally published	Yes
Event	31th European Conference on Information Retrieval, ECIR 2009 - Toulouse, France Duration: Apr 6 2009 → Apr 9 2009

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	5478 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Other

Other	31th European Conference on Information Retrieval, ECIR 2009
Country/Territory	France
City	Toulouse
Period	4/6/09 → 4/9/09

ASJC Scopus subject areas

Theoretical Computer Science
General Computer Science

Online availability

10.1007/978-3-642-00958-7_28

Library availability

Discover UIUC Full Text

Cite this

Efron, M. (2009). Using multiple uery aspects to build test collections without human relevance udgments. In Advances in Information Retrieval - 31th European Conference on IR Research, ECIR 2009, Proceedings (pp. 301-312). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 5478 LNCS). https://doi.org/10.1007/978-3-642-00958-7_28

Using multiple uery aspects to build test collections without human relevance udgments. / Efron, Miles.
Advances in Information Retrieval - 31th European Conference on IR Research, ECIR 2009, Proceedings. 2009. p. 301-312 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 5478 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Efron, M 2009, Using multiple uery aspects to build test collections without human relevance udgments. in Advances in Information Retrieval - 31th European Conference on IR Research, ECIR 2009, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 5478 LNCS, pp. 301-312, 31th European Conference on Information Retrieval, ECIR 2009, Toulouse, France, 4/6/09. https://doi.org/10.1007/978-3-642-00958-7_28

Efron M. Using multiple uery aspects to build test collections without human relevance udgments. In Advances in Information Retrieval - 31th European Conference on IR Research, ECIR 2009, Proceedings. 2009. p. 301-312. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-642-00958-7_28

@inproceedings{ced309a65fb548159e600bc43098745f,

title = "Using multiple uery aspects to build test collections without human relevance udgments",

abstract = "Collecting relevance judgments (qrels) is an especially challenging part of building an information retrieval test collection. This paper presents a novel method for creating test collections by offering a substitute for relevance judgments. Our method is based on an old idea in IR: a single information need can be represented by many query articulations. We call different articulations of a particular need query aspects. By combining the top k documents retrieved by a single system for multiple query aspects, we build judgment-free qrels whose rank ordering of IR systems correlates highly with rankings based on human relevance judgments.",

author = "Miles Efron",

year = "2009",

doi = "10.1007/978-3-642-00958-7_28",

language = "English (US)",

isbn = "3642009573",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

pages = "301--312",

booktitle = "Advances in Information Retrieval - 31th European Conference on IR Research, ECIR 2009, Proceedings",

note = "31th European Conference on Information Retrieval, ECIR 2009 ; Conference date: 06-04-2009 Through 09-04-2009",

}

TY - GEN

T1 - Using multiple uery aspects to build test collections without human relevance udgments

AU - Efron, Miles

PY - 2009

Y1 - 2009

N2 - Collecting relevance judgments (qrels) is an especially challenging part of building an information retrieval test collection. This paper presents a novel method for creating test collections by offering a substitute for relevance judgments. Our method is based on an old idea in IR: a single information need can be represented by many query articulations. We call different articulations of a particular need query aspects. By combining the top k documents retrieved by a single system for multiple query aspects, we build judgment-free qrels whose rank ordering of IR systems correlates highly with rankings based on human relevance judgments.

AB - Collecting relevance judgments (qrels) is an especially challenging part of building an information retrieval test collection. This paper presents a novel method for creating test collections by offering a substitute for relevance judgments. Our method is based on an old idea in IR: a single information need can be represented by many query articulations. We call different articulations of a particular need query aspects. By combining the top k documents retrieved by a single system for multiple query aspects, we build judgment-free qrels whose rank ordering of IR systems correlates highly with rankings based on human relevance judgments.

UR - http://www.scopus.com/inward/record.url?scp=67650705964&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=67650705964&partnerID=8YFLogxK

U2 - 10.1007/978-3-642-00958-7_28

DO - 10.1007/978-3-642-00958-7_28

M3 - Conference contribution

AN - SCOPUS:67650705964

SN - 3642009573

SN - 9783642009570

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 301

EP - 312

BT - Advances in Information Retrieval - 31th European Conference on IR Research, ECIR 2009, Proceedings

T2 - 31th European Conference on Information Retrieval, ECIR 2009

Y2 - 6 April 2009 through 9 April 2009

ER -

Using multiple uery aspects to build test collections without human relevance udgments

Abstract

Publication series

Other

ASJC Scopus subject areas

Online availability

Library availability

Related links

Fingerprint

Cite this