TY - JOUR

T1 - Mean field analysis of neural networks

T2 - A central limit theorem

AU - Sirignano, Justin

AU - Spiliopoulos, Konstantinos

N1 - Funding Information:
K.S. was partially supported by the National Science Foundation, United States (DMS 1550918).
Publisher Copyright:
© 2019 Elsevier B.V.

PY - 2020/3

Y1 - 2020/3

N2 - We rigorously prove a central limit theorem for neural network models with a single hidden layer. The central limit theorem is proven in the asymptotic regime of simultaneously (A) large numbers of hidden units and (B) large numbers of stochastic gradient descent training iterations. Our result describes the neural network's fluctuations around its mean-field limit. The fluctuations have a Gaussian distribution and satisfy a stochastic partial differential equation. The proof relies upon weak convergence methods from stochastic analysis. In particular, we prove relative compactness for the sequence of processes and uniqueness of the limiting process in a suitable Sobolev space.

AB - We rigorously prove a central limit theorem for neural network models with a single hidden layer. The central limit theorem is proven in the asymptotic regime of simultaneously (A) large numbers of hidden units and (B) large numbers of stochastic gradient descent training iterations. Our result describes the neural network's fluctuations around its mean-field limit. The fluctuations have a Gaussian distribution and satisfy a stochastic partial differential equation. The proof relies upon weak convergence methods from stochastic analysis. In particular, we prove relative compactness for the sequence of processes and uniqueness of the limiting process in a suitable Sobolev space.

UR - http://www.scopus.com/inward/record.url?scp=85068268926&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85068268926&partnerID=8YFLogxK

U2 - 10.1016/j.spa.2019.06.003

DO - 10.1016/j.spa.2019.06.003

M3 - Article

AN - SCOPUS:85068268926

SN - 0304-4149

VL - 130

SP - 1820

EP - 1852

JO - Stochastic Processes and their Applications

JF - Stochastic Processes and their Applications

IS - 3

ER -