TY - GEN
T1 - Tightening Mutual Information Based Bounds on Generalization Error
AU - Bu, Yuheng
AU - Zou, Shaofeng
AU - Veeravalli, Venugopal V.
N1 - Publisher Copyright:
© 2019 IEEE.
PY - 2019/7
Y1 - 2019/7
N2 - A mutual information based upper bound on the generalization error of a supervised learning algorithm is derived in this paper. The bound is constructed in terms of the mutual information between each individual training sample and the output of the learning algorithm, which requires weaker conditions on the loss function, but provides a tighter characterization of the generalization error than existing studies. Examples are further provided to demonstrate that the bound derived in this paper is tighter, and has a broader range of applicability. Application to noisy and iterative algorithms, e.g., stochastic gradient Langevin dynamics (SGLD), is also studied, where the constructed bound provides a tighter characterization of the generalization error than existing results.
AB - A mutual information based upper bound on the generalization error of a supervised learning algorithm is derived in this paper. The bound is constructed in terms of the mutual information between each individual training sample and the output of the learning algorithm, which requires weaker conditions on the loss function, but provides a tighter characterization of the generalization error than existing studies. Examples are further provided to demonstrate that the bound derived in this paper is tighter, and has a broader range of applicability. Application to noisy and iterative algorithms, e.g., stochastic gradient Langevin dynamics (SGLD), is also studied, where the constructed bound provides a tighter characterization of the generalization error than existing results.
UR - http://www.scopus.com/inward/record.url?scp=85073167760&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85073167760&partnerID=8YFLogxK
U2 - 10.1109/ISIT.2019.8849590
DO - 10.1109/ISIT.2019.8849590
M3 - Conference contribution
AN - SCOPUS:85073167760
T3 - IEEE International Symposium on Information Theory - Proceedings
SP - 587
EP - 591
BT - 2019 IEEE International Symposium on Information Theory, ISIT 2019 - Proceedings
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 2019 IEEE International Symposium on Information Theory, ISIT 2019
Y2 - 7 July 2019 through 12 July 2019
ER -