RATE: A Reliability-Aware Tester-Based Evaluation Framework of User Simulators

Sahiti Labhishetty; Cheng Xiang Zhai

doi:10.1007/978-3-030-99736-6_23

RATE: A Reliability-Aware Tester-Based Evaluation Framework of User Simulators

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Evaluation of user simulators is needed in order to use them for evaluating Interactive Information Retrieval (IIR) Systems. Previous work has proposed a tester-based approach to evaluate user simulators, but it has not addressed the important question about the reliability of the testers themselves, nor has it studied how to generate a single reliability score for a user simulator based on multiple testers. In this paper, we address these two limitations and propose a novel Reliability-Aware Tester-based Evaluation (RATE) framework for evaluating the reliability of both User Simulators and testers. In this framework, the reliability of Testers and that of Simulators are jointly learned through unsupervised learning using iterative propagation of reliability. We propose and evaluate two algorithms for unsupervised learning of reliabilities. Evaluation results using TREC data sets show that the proposed RATE framework is effective in measuring the reliability of simulators and testers, thus serving as a foundation for potentially establishing a new paradigm for evaluating IIR systems using user simulation.

Original language	English (US)
Title of host publication	Advances in Information Retrieval - 44th European Conference on IR Research, ECIR 2022, Proceedings
Editors	Matthias Hagen, Suzan Verberne, Craig Macdonald, Christin Seifert, Krisztian Balog, Kjetil Nørvåg, Vinay Setty
Publisher	Springer
Pages	336-350
Number of pages	15
ISBN (Print)	9783030997359
DOIs	https://doi.org/10.1007/978-3-030-99736-6_23
State	Published - 2022
Event	44th European Conference on Information Retrieval, ECIR 2022 - Stavanger, Norway Duration: Apr 10 2022 → Apr 14 2022

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	13185 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	44th European Conference on Information Retrieval, ECIR 2022
Country/Territory	Norway
City	Stavanger
Period	4/10/22 → 4/14/22

Keywords

IIR Systems
Reliability of User Simulator
Tester

ASJC Scopus subject areas

Theoretical Computer Science
General Computer Science

Online availability

10.1007/978-3-030-99736-6_23

Library availability

Discover UIUC Full Text

Cite this

Labhishetty, S., & Zhai, C. X. (2022). RATE: A Reliability-Aware Tester-Based Evaluation Framework of User Simulators. In M. Hagen, S. Verberne, C. Macdonald, C. Seifert, K. Balog, K. Nørvåg, & V. Setty (Eds.), Advances in Information Retrieval - 44th European Conference on IR Research, ECIR 2022, Proceedings (pp. 336-350). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13185 LNCS). Springer. https://doi.org/10.1007/978-3-030-99736-6_23

RATE: A Reliability-Aware Tester-Based Evaluation Framework of User Simulators. / Labhishetty, Sahiti; Zhai, Cheng Xiang.
Advances in Information Retrieval - 44th European Conference on IR Research, ECIR 2022, Proceedings. ed. / Matthias Hagen; Suzan Verberne; Craig Macdonald; Christin Seifert; Krisztian Balog; Kjetil Nørvåg; Vinay Setty. Springer, 2022. p. 336-350 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13185 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Labhishetty, S & Zhai, CX 2022, RATE: A Reliability-Aware Tester-Based Evaluation Framework of User Simulators. in M Hagen, S Verberne, C Macdonald, C Seifert, K Balog, K Nørvåg & V Setty (eds), Advances in Information Retrieval - 44th European Conference on IR Research, ECIR 2022, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13185 LNCS, Springer, pp. 336-350, 44th European Conference on Information Retrieval, ECIR 2022, Stavanger, Norway, 4/10/22. https://doi.org/10.1007/978-3-030-99736-6_23

Labhishetty S, Zhai CX. RATE: A Reliability-Aware Tester-Based Evaluation Framework of User Simulators. In Hagen M, Verberne S, Macdonald C, Seifert C, Balog K, Nørvåg K, Setty V, editors, Advances in Information Retrieval - 44th European Conference on IR Research, ECIR 2022, Proceedings. Springer. 2022. p. 336-350. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-99736-6_23

Labhishetty, Sahiti ; Zhai, Cheng Xiang. / RATE : A Reliability-Aware Tester-Based Evaluation Framework of User Simulators. Advances in Information Retrieval - 44th European Conference on IR Research, ECIR 2022, Proceedings. editor / Matthias Hagen ; Suzan Verberne ; Craig Macdonald ; Christin Seifert ; Krisztian Balog ; Kjetil Nørvåg ; Vinay Setty. Springer, 2022. pp. 336-350 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{c5b9db4376924432afde463079c481b1,

title = "RATE: A Reliability-Aware Tester-Based Evaluation Framework of User Simulators",

abstract = "Evaluation of user simulators is needed in order to use them for evaluating Interactive Information Retrieval (IIR) Systems. Previous work has proposed a tester-based approach to evaluate user simulators, but it has not addressed the important question about the reliability of the testers themselves, nor has it studied how to generate a single reliability score for a user simulator based on multiple testers. In this paper, we address these two limitations and propose a novel Reliability-Aware Tester-based Evaluation (RATE) framework for evaluating the reliability of both User Simulators and testers. In this framework, the reliability of Testers and that of Simulators are jointly learned through unsupervised learning using iterative propagation of reliability. We propose and evaluate two algorithms for unsupervised learning of reliabilities. Evaluation results using TREC data sets show that the proposed RATE framework is effective in measuring the reliability of simulators and testers, thus serving as a foundation for potentially establishing a new paradigm for evaluating IIR systems using user simulation.",

keywords = "IIR Systems, Reliability of User Simulator, Tester",

author = "Sahiti Labhishetty and Zhai, {Cheng Xiang}",

note = "Publisher Copyright: {\textcopyright} 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.; 44th European Conference on Information Retrieval, ECIR 2022 ; Conference date: 10-04-2022 Through 14-04-2022",

year = "2022",

doi = "10.1007/978-3-030-99736-6_23",

language = "English (US)",

isbn = "9783030997359",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer",

pages = "336--350",

editor = "Matthias Hagen and Suzan Verberne and Craig Macdonald and Christin Seifert and Krisztian Balog and Kjetil N{\o}rv{\aa}g and Vinay Setty",

booktitle = "Advances in Information Retrieval - 44th European Conference on IR Research, ECIR 2022, Proceedings",

address = "Germany",

}

TY - GEN

T1 - RATE

T2 - 44th European Conference on Information Retrieval, ECIR 2022

AU - Labhishetty, Sahiti

AU - Zhai, Cheng Xiang

PY - 2022

Y1 - 2022

N2 - Evaluation of user simulators is needed in order to use them for evaluating Interactive Information Retrieval (IIR) Systems. Previous work has proposed a tester-based approach to evaluate user simulators, but it has not addressed the important question about the reliability of the testers themselves, nor has it studied how to generate a single reliability score for a user simulator based on multiple testers. In this paper, we address these two limitations and propose a novel Reliability-Aware Tester-based Evaluation (RATE) framework for evaluating the reliability of both User Simulators and testers. In this framework, the reliability of Testers and that of Simulators are jointly learned through unsupervised learning using iterative propagation of reliability. We propose and evaluate two algorithms for unsupervised learning of reliabilities. Evaluation results using TREC data sets show that the proposed RATE framework is effective in measuring the reliability of simulators and testers, thus serving as a foundation for potentially establishing a new paradigm for evaluating IIR systems using user simulation.

AB - Evaluation of user simulators is needed in order to use them for evaluating Interactive Information Retrieval (IIR) Systems. Previous work has proposed a tester-based approach to evaluate user simulators, but it has not addressed the important question about the reliability of the testers themselves, nor has it studied how to generate a single reliability score for a user simulator based on multiple testers. In this paper, we address these two limitations and propose a novel Reliability-Aware Tester-based Evaluation (RATE) framework for evaluating the reliability of both User Simulators and testers. In this framework, the reliability of Testers and that of Simulators are jointly learned through unsupervised learning using iterative propagation of reliability. We propose and evaluate two algorithms for unsupervised learning of reliabilities. Evaluation results using TREC data sets show that the proposed RATE framework is effective in measuring the reliability of simulators and testers, thus serving as a foundation for potentially establishing a new paradigm for evaluating IIR systems using user simulation.

KW - IIR Systems

KW - Reliability of User Simulator

KW - Tester

UR - http://www.scopus.com/inward/record.url?scp=85128722131&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85128722131&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-99736-6_23

DO - 10.1007/978-3-030-99736-6_23

M3 - Conference contribution

AN - SCOPUS:85128722131

SN - 9783030997359

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 336

EP - 350

BT - Advances in Information Retrieval - 44th European Conference on IR Research, ECIR 2022, Proceedings

A2 - Hagen, Matthias

A2 - Verberne, Suzan

A2 - Macdonald, Craig

A2 - Seifert, Christin

A2 - Balog, Krisztian

A2 - Nørvåg, Kjetil

A2 - Setty, Vinay

PB - Springer

Y2 - 10 April 2022 through 14 April 2022

ER -