VIR Lab: A platform for privacy-preserving evaluation for information retrieval models

Information retrieval (IR) has been a highly empirical discipline since the very beginning of the field. The development and study of any novel techniques such as retrieval models always require extensive experiments over multiple representative data collections. Traditionally, IR evaluation relies on the use of publicly available data, so researchers often download the collections and conduct the evaluation on their servers. However, this would not be a favorable (or even possible) solution to evaluation over the proprietary data due to various privacy concerns. In this paper, we discuss one potential solution to the privacy-preserving evaluation (PPE) for IR models. We first briefly introduce the VIRLab system, and then discuss how to extend the system to enable a controlled data-centric experimental environment for evaluation over proprietary data.