Mean field analysis of neural networks: A law of large numbers

Justin Sirignano, Konstantinos Spiliopoulos

Research output: Contribution to journalArticlepeer-review

Abstract

Machine learning, and in particular neural network models, have revolutionized fields such as image, text, and speech recognition. Today, many important real-world applications in these areas are driven by neural networks. There are also growing applications in engineering, robotics, medicine, and finance. Despite their immense success in practice, there is limited mathematical understanding of neural networks. This paper illustrates how neural networks can be studied via stochastic analysis and develops approaches for addressing some of the technical challenges which arise. We analyze one-layer neural networks in the asymptotic regime of simultaneously (a) large network sizes and (b) large numbers of stochastic gradient descent training iterations. We rigorously prove that the empirical distribution of the neural network parameters converges to the solution of a nonlinear partial differential equation. This result can be considered a law of large numbers for neural networks. In addition, a consequence of our analysis is that the trained parameters of the neural network asymptotically become independent, a property which is commonly called "propagation of chaos.

Original languageEnglish (US)
Pages (from-to)725-752
Number of pages28
JournalSIAM Journal on Applied Mathematics
Volume80
Issue number2
DOIs
StatePublished - 2020

Keywords

  • Machine learning
  • Stochastic analysis
  • Weak convergence

ASJC Scopus subject areas

  • Applied Mathematics

Fingerprint Dive into the research topics of 'Mean field analysis of neural networks: A law of large numbers'. Together they form a unique fingerprint.

Cite this