Information-theoretic generalization bounds for black-box learning algorithms

Hrayr Harutyunyan; Maxim Raginsky; Greg Ver Steeg; Aram Galstyan

Information-theoretic generalization bounds for black-box learning algorithms

Hrayr Harutyunyan, Maxim Raginsky, Greg Ver Steeg, Aram Galstyan

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

We derive information-theoretic generalization bounds for supervised learning algorithms based on the information contained in predictions rather than in the output of the training algorithm. These bounds improve over the existing information-theoretic bounds, are applicable to a wider range of algorithms, and solve two key challenges: (a) they give meaningful results for deterministic algorithms and (b) they are significantly easier to estimate. We show experimentally that the proposed bounds closely follow the generalization gap in practical scenarios for deep learning.

Original language	English (US)
Title of host publication	Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021
Editors	Marc'Aurelio Ranzato, Alina Beygelzimer, Yann Dauphin, Percy S. Liang, Jenn Wortman Vaughan
Publisher	Neural information processing systems foundation
Pages	24670-24682
Number of pages	13
ISBN (Electronic)	9781713845393
State	Published - 2021
Event	35th Conference on Neural Information Processing Systems, NeurIPS 2021 - Virtual, Online Duration: Dec 6 2021 → Dec 14 2021

Publication series

Name	Advances in Neural Information Processing Systems
Volume	30
ISSN (Print)	1049-5258

Conference

Conference	35th Conference on Neural Information Processing Systems, NeurIPS 2021
City	Virtual, Online
Period	12/6/21 → 12/14/21

ASJC Scopus subject areas

Computer Networks and Communications
Information Systems
Signal Processing

Library availability

Discover UIUC Full Text

Cite this

Harutyunyan, H., Raginsky, M., Steeg, G. V., & Galstyan, A. (2021). Information-theoretic generalization bounds for black-box learning algorithms. In MA. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, & J. Wortman Vaughan (Eds.), Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021 (pp. 24670-24682). (Advances in Neural Information Processing Systems; Vol. 30). Neural information processing systems foundation.

Information-theoretic generalization bounds for black-box learning algorithms. / Harutyunyan, Hrayr; Raginsky, Maxim; Steeg, Greg Ver et al.
Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021. ed. / Marc'Aurelio Ranzato; Alina Beygelzimer; Yann Dauphin; Percy S. Liang; Jenn Wortman Vaughan. Neural information processing systems foundation, 2021. p. 24670-24682 (Advances in Neural Information Processing Systems; Vol. 30).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Harutyunyan, H, Raginsky, M, Steeg, GV & Galstyan, A 2021, Information-theoretic generalization bounds for black-box learning algorithms. in MA Ranzato, A Beygelzimer, Y Dauphin, PS Liang & J Wortman Vaughan (eds), Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021. Advances in Neural Information Processing Systems, vol. 30, Neural information processing systems foundation, pp. 24670-24682, 35th Conference on Neural Information Processing Systems, NeurIPS 2021, Virtual, Online, 12/6/21.

Harutyunyan H, Raginsky M, Steeg GV, Galstyan A. Information-theoretic generalization bounds for black-box learning algorithms. In Ranzato MA, Beygelzimer A, Dauphin Y, Liang PS, Wortman Vaughan J, editors, Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021. Neural information processing systems foundation. 2021. p. 24670-24682. (Advances in Neural Information Processing Systems).

Harutyunyan, Hrayr ; Raginsky, Maxim ; Steeg, Greg Ver et al. / Information-theoretic generalization bounds for black-box learning algorithms. Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021. editor / Marc'Aurelio Ranzato ; Alina Beygelzimer ; Yann Dauphin ; Percy S. Liang ; Jenn Wortman Vaughan. Neural information processing systems foundation, 2021. pp. 24670-24682 (Advances in Neural Information Processing Systems).

@inproceedings{e7fc3c9fd70a435dbe579855d90528bd,

title = "Information-theoretic generalization bounds for black-box learning algorithms",

abstract = "We derive information-theoretic generalization bounds for supervised learning algorithms based on the information contained in predictions rather than in the output of the training algorithm. These bounds improve over the existing information-theoretic bounds, are applicable to a wider range of algorithms, and solve two key challenges: (a) they give meaningful results for deterministic algorithms and (b) they are significantly easier to estimate. We show experimentally that the proposed bounds closely follow the generalization gap in practical scenarios for deep learning.",

author = "Hrayr Harutyunyan and Maxim Raginsky and Steeg, {Greg Ver} and Aram Galstyan",

note = "Publisher Copyright: {\textcopyright} 2021 Neural information processing systems foundation. All rights reserved.; 35th Conference on Neural Information Processing Systems, NeurIPS 2021 ; Conference date: 06-12-2021 Through 14-12-2021",

year = "2021",

language = "English (US)",

series = "Advances in Neural Information Processing Systems",

publisher = "Neural information processing systems foundation",

pages = "24670--24682",

editor = "Marc'Aurelio Ranzato and Alina Beygelzimer and Yann Dauphin and Liang, {Percy S.} and {Wortman Vaughan}, Jenn",

booktitle = "Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021",

}

TY - GEN

T1 - Information-theoretic generalization bounds for black-box learning algorithms

AU - Harutyunyan, Hrayr

AU - Raginsky, Maxim

AU - Steeg, Greg Ver

AU - Galstyan, Aram

PY - 2021

Y1 - 2021

N2 - We derive information-theoretic generalization bounds for supervised learning algorithms based on the information contained in predictions rather than in the output of the training algorithm. These bounds improve over the existing information-theoretic bounds, are applicable to a wider range of algorithms, and solve two key challenges: (a) they give meaningful results for deterministic algorithms and (b) they are significantly easier to estimate. We show experimentally that the proposed bounds closely follow the generalization gap in practical scenarios for deep learning.

AB - We derive information-theoretic generalization bounds for supervised learning algorithms based on the information contained in predictions rather than in the output of the training algorithm. These bounds improve over the existing information-theoretic bounds, are applicable to a wider range of algorithms, and solve two key challenges: (a) they give meaningful results for deterministic algorithms and (b) they are significantly easier to estimate. We show experimentally that the proposed bounds closely follow the generalization gap in practical scenarios for deep learning.

UR - http://www.scopus.com/inward/record.url?scp=85130809906&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85130809906&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85130809906

T3 - Advances in Neural Information Processing Systems

SP - 24670

EP - 24682

BT - Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021

A2 - Ranzato, Marc'Aurelio

A2 - Beygelzimer, Alina

A2 - Dauphin, Yann

A2 - Liang, Percy S.

A2 - Wortman Vaughan, Jenn

PB - Neural information processing systems foundation

T2 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021

Y2 - 6 December 2021 through 14 December 2021

ER -

Information-theoretic generalization bounds for black-box learning algorithms

Abstract

Publication series

Conference

ASJC Scopus subject areas

Library availability

Related links

Fingerprint

Cite this