TY - GEN
T1 - Analysis of adaptive training for learning to rank in information retrieval
AU - Kuzi, Saar
AU - Labhishetty, Sahiti
AU - Santu, Shubhra Kanti Karmaker
AU - Joshi, Prasad Pradip
AU - Zhai, Cheng Xiang
N1 - Funding Information:
In this paper, we have done an in-depth empirical analysis of the benefit and challenge of adaptive training for LTR. We proposed a novel Best-Feature Calibration (BFC) strategy for analysis of LTR and used it to analyze the clustering-based adaptive training approach. Our study reveals that LTR, when trained on the entire data, can substantially perform worse than the best single feature on many queries, suggesting that the robustness of LTR is a serious concern in search engines as it can potentially negatively impact the satisfaction of users. Our analysis further shows that the main benefit of the clustering-based adaptive training is in reducing the performance reduction of LTR when it performed worse than the best single feature, and thus is a beneficial strategy that can be adopted in search engines to improve the robustness of LTR. Finally, we further provided an explanation of why fusion of clusters tends to be more robust and proposed an error decomposition framework for analyzing the errors of this approach. Acknowledgments. This material is based upon work supported by Unbxd and the National Science Foundation under Grant No. 1801652.
Publisher Copyright:
© 2019 Association for Computing Machinery.
PY - 2019/11/3
Y1 - 2019/11/3
N2 - Learning to Rank is an important framework used in search engines to optimize the combination of multiple features in a single ranking function. In the existing work on learning to rank, such a ranking function is often trained on a large set of different queries to optimize the overall performance on all of them. However, the optimal parameters to combine those features are generally query-dependent, making such a strategy of “one size fits all” non-optimal. Some previous works have addressed this problem by suggesting a query-level adaptive training for learning to rank with promising results. However, previous work has not analyzed the reasons for the improvement. In this paper, we present a Best-Feature Calibration (BFC) strategy for analyzing learning to rank models and use this strategy to examine the benefit of query-level adaptive training. Our results show that the benefit of adaptive training mainly lies in the improvement of the robustness of learning to rank in cases where it does not perform as well as the best single feature.
AB - Learning to Rank is an important framework used in search engines to optimize the combination of multiple features in a single ranking function. In the existing work on learning to rank, such a ranking function is often trained on a large set of different queries to optimize the overall performance on all of them. However, the optimal parameters to combine those features are generally query-dependent, making such a strategy of “one size fits all” non-optimal. Some previous works have addressed this problem by suggesting a query-level adaptive training for learning to rank with promising results. However, previous work has not analyzed the reasons for the improvement. In this paper, we present a Best-Feature Calibration (BFC) strategy for analyzing learning to rank models and use this strategy to examine the benefit of query-level adaptive training. Our results show that the benefit of adaptive training mainly lies in the improvement of the robustness of learning to rank in cases where it does not perform as well as the best single feature.
UR - http://www.scopus.com/inward/record.url?scp=85075451594&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85075451594&partnerID=8YFLogxK
U2 - 10.1145/3357384.3358159
DO - 10.1145/3357384.3358159
M3 - Conference contribution
AN - SCOPUS:85075451594
T3 - International Conference on Information and Knowledge Management, Proceedings
SP - 2325
EP - 2328
BT - CIKM 2019 - Proceedings of the 28th ACM International Conference on Information and Knowledge Management
PB - Association for Computing Machinery
T2 - 28th ACM International Conference on Information and Knowledge Management, CIKM 2019
Y2 - 3 November 2019 through 7 November 2019
ER -