Hessian based analysis of SGD for Deep Nets: Dynamics and generalization

Xinyan Li, Qilong Gu, Yingxue Zhou, Tiancong Chen, Arindam Banerjee

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

While stochastic gradient descent (SGD) and variants have been surprisingly successful for training deep nets, several aspects of the optimization dynamics and generalization are still not well understood. In this paper, we present new empirical observations and theoretical results on both the optimization dynamics and generalization behavior of SGD for deep nets based on the Hessian of the training loss and associated quantities. We consider three specific research questions: (1) what is the relationship between the Hessian of the loss and the second moment of stochastic gradients (SGs)? (2) how can we characterize the stochastic optimization dynamics of SGD with fixed step sizes based on the first and second moments of SGs? and (3) how can we characterize a scale-invariant generalization bound of deep nets based on the Hessian of the loss? Throughout the paper, we support theoretical results with empirical observations, with experiments on synthetic data, MNIST, and CIFAR-10, with different batch sizes, and with different difficulty levels by synthetically adding random labels.

Original languageEnglish (US)
Title of host publicationProceedings of the 2020 SIAM International Conference on Data Mining, SDM 2020
EditorsCarlotta Demeniconi, Nitesh Chawla
PublisherSociety for Industrial and Applied Mathematics Publications
Pages190-198
Number of pages9
ISBN (Electronic)9781611976236
DOIs
StatePublished - 2020
Externally publishedYes
Event2020 SIAM International Conference on Data Mining, SDM 2020 - Cincinnati, United States
Duration: May 7 2020May 9 2020

Publication series

NameProceedings of the 2020 SIAM International Conference on Data Mining, SDM 2020

Conference

Conference2020 SIAM International Conference on Data Mining, SDM 2020
CountryUnited States
CityCincinnati
Period5/7/205/9/20

ASJC Scopus subject areas

  • Computer Science Applications
  • Software

Fingerprint Dive into the research topics of 'Hessian based analysis of SGD for Deep Nets: Dynamics and generalization'. Together they form a unique fingerprint.

Cite this