TY - GEN
T1 - Consistency analysis for binary classification revisited
AU - Dembczyriski, Krzysztof
AU - Kottowski, Wojciech
AU - Koyejo, Oluwasanmi
AU - Natarajan, Nagarajan
N1 - Publisher Copyright:
Copyright 2017 by the author(s).
PY - 2017
Y1 - 2017
N2 - Statistical learning theory is at an inflection point enabled by recent advances in understanding and optimizing a wide range of metrics. Of particular interest are non-decomposable metrics such as the F-measure and the Jaccard measure which cannot be represented as a simple average over examples. Non-decomposability is the primary source of difficulty in theoretical analysis, and interestingly has led to two distinct settings and notions of consistency. In this manuscript we analyze both settings, from statistical and algorithmic points of view, to explore the connections and to highlight differences between them for a wide range of metrics. The analysis complements previous results on this topic, clarifies common confusions around both settings, and provides guidance to the theory and practice of binary classification with complex metrics.
AB - Statistical learning theory is at an inflection point enabled by recent advances in understanding and optimizing a wide range of metrics. Of particular interest are non-decomposable metrics such as the F-measure and the Jaccard measure which cannot be represented as a simple average over examples. Non-decomposability is the primary source of difficulty in theoretical analysis, and interestingly has led to two distinct settings and notions of consistency. In this manuscript we analyze both settings, from statistical and algorithmic points of view, to explore the connections and to highlight differences between them for a wide range of metrics. The analysis complements previous results on this topic, clarifies common confusions around both settings, and provides guidance to the theory and practice of binary classification with complex metrics.
UR - http://www.scopus.com/inward/record.url?scp=85048380409&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85048380409&partnerID=8YFLogxK
M3 - Conference contribution
AN - SCOPUS:85048380409
T3 - 34th International Conference on Machine Learning, ICML 2017
SP - 1597
EP - 1613
BT - 34th International Conference on Machine Learning, ICML 2017
PB - International Machine Learning Society (IMLS)
T2 - 34th International Conference on Machine Learning, ICML 2017
Y2 - 6 August 2017 through 11 August 2017
ER -