Infrastructure support for evaluation as a service

Jimmy Lin, Miles Efron

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

How do we conduct large-scale community-wide evaluations for information retrieval if we are unable to distribute the document collection? This was the challenge we faced in organizing a task on searching tweets at the Text Retrieval Conference (TREC), since Twitter's terms of service forbid redistribution of tweets. Our solution, which we call eval- uation as a service", was to provide an API through which the collection can be accessed for completing the evaluation task. This paper describes the infrastructure underlying the service and its deployment at TREC 2013. We discuss the merits of the approach and potential applicability to other evaluation scenarios.

Original languageEnglish (US)
Title of host publicationWWW 2014 Companion - Proceedings of the 23rd International Conference on World Wide Web
PublisherAssociation for Computing Machinery, Inc
Pages79-82
Number of pages4
ISBN (Electronic)9781450327459
DOIs
StatePublished - Apr 7 2014
Event23rd International Conference on World Wide Web, WWW 2014 - Seoul, Korea, Republic of
Duration: Apr 7 2014Apr 11 2014

Publication series

NameWWW 2014 Companion - Proceedings of the 23rd International Conference on World Wide Web

Other

Other23rd International Conference on World Wide Web, WWW 2014
CountryKorea, Republic of
CitySeoul
Period4/7/144/11/14

Keywords

  • TREC microblog
  • Tweet search

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Software

Fingerprint Dive into the research topics of 'Infrastructure support for evaluation as a service'. Together they form a unique fingerprint.

Cite this