TY - GEN
T1 - Quadratic Metric Elicitation for Fairness and Beyond
AU - Hiranandani, Gaurush
AU - Mathur, Jatin
AU - Narasimhan, Harikrishna
AU - Koyejo, Oluwasanmi
N1 - Funding Information:
This research was funded by Google Research. The authors would like to thank Safinah Ali, Sohini Upadhyay, and Elena Glassman for helping with the pilot user study discussed in Appendix J.
Publisher Copyright:
© 2022 Proceedings of the 38th Conference on Uncertainty in Artificial Intelligence, UAI 2022. All right reserved.
PY - 2022
Y1 - 2022
N2 - Metric elicitation is a recent framework for eliciting classification performance metrics that best reflect implicit user preferences based on the task and context. However, available elicitation strategies have been limited to linear (or quasi-linear) functions of predictive rates, which can be practically restrictive for many applications including fairness. This paper develops a strategy for eliciting more flexible multiclass metrics defined by quadratic functions of rates, designed to reflect human preferences better. We show its application in eliciting quadratic violation-based group-fair metrics. Our strategy requires only relative preference feedback, is robust to noise, and achieves near-optimal query complexity. We further extend this strategy to eliciting polynomial metrics - thus broadening the use cases for metric elicitation.
AB - Metric elicitation is a recent framework for eliciting classification performance metrics that best reflect implicit user preferences based on the task and context. However, available elicitation strategies have been limited to linear (or quasi-linear) functions of predictive rates, which can be practically restrictive for many applications including fairness. This paper develops a strategy for eliciting more flexible multiclass metrics defined by quadratic functions of rates, designed to reflect human preferences better. We show its application in eliciting quadratic violation-based group-fair metrics. Our strategy requires only relative preference feedback, is robust to noise, and achieves near-optimal query complexity. We further extend this strategy to eliciting polynomial metrics - thus broadening the use cases for metric elicitation.
UR - http://www.scopus.com/inward/record.url?scp=85144282602&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85144282602&partnerID=8YFLogxK
M3 - Conference contribution
AN - SCOPUS:85144282602
T3 - Proceedings of the 38th Conference on Uncertainty in Artificial Intelligence, UAI 2022
SP - 811
EP - 821
BT - Proceedings of the 38th Conference on Uncertainty in Artificial Intelligence, UAI 2022
PB - Association For Uncertainty in Artificial Intelligence (AUAI)
T2 - 38th Conference on Uncertainty in Artificial Intelligence, UAI 2022
Y2 - 1 August 2022 through 5 August 2022
ER -