Neural Tangent Kernel at Initialization: Linear Width Suffices

Arindam Banerjee, Pedro Cisneros-Velarde, Libin Zhu, Mikhail Belkin

Research output: Contribution to journalConference articlepeer-review


In this paper we study the problem of lower bounding the minimum eigenvalue of the neural tangent kernel (NTK) at initialization, an important quantity for the theoretical analysis of training in neural networks. We consider feedforward neural networks with smooth activation functions. Without any distributional assumptions on the input, we present a novel result: we show that for suitable initialization variance, Ω(e n) width, where n is the number of training samples, suffices to ensure that the NTK at initialization is positive definite, improving prior results for smooth activations under our setting. Prior to our work, the sufficiency of linear width has only been shown either for networks with ReLU activation functions, and sublinear width has been shown for smooth networks but with additional conditions on the distribution of the data. The technical challenge in the analysis stems from the layerwise inhomogeneity of smooth activation functions and we handle the challenge using generalized Hermite series expansion of such activations.

Original languageEnglish (US)
Pages (from-to)110-118
Number of pages9
JournalProceedings of Machine Learning Research
StatePublished - 2023
Event39th Conference on Uncertainty in Artificial Intelligence, UAI 2023 - Pittsburgh, United States
Duration: Jul 31 2023Aug 4 2023

ASJC Scopus subject areas

  • Artificial Intelligence
  • Software
  • Control and Systems Engineering
  • Statistics and Probability


Dive into the research topics of 'Neural Tangent Kernel at Initialization: Linear Width Suffices'. Together they form a unique fingerprint.

Cite this