Understanding the loss surface of single-layered neural networks for binary classification

Shiyu Liang, Ruoyu Sun, Rayadurgam Srikant, Yixuan Li

Research output: Contribution to conferencePaper

Abstract

It is widely conjectured that the reason that training algorithms for neural networks are successful because all local minima lead to similar performance; for example, see (LeCun et al., 2015; Choromanska et al., 2015; Dauphin et al., 2014). Performance is typically measured in terms of two metrics: training performance and generalization performance. Here we focus on the training performance of single-layered neural networks for binary classification, and provide conditions under which the training error is zero at all local minima of a smooth hinge loss function. Our conditions are roughly in the following form: the neurons have to be strictly convex and the surrogate loss function should be a smooth version of hinge loss. We also provide counterexamples to show that when the loss function is replaced with quadratic loss or logistic loss, the result may not hold.

Original languageEnglish (US)
StatePublished - Jan 1 2018
Event6th International Conference on Learning Representations, ICLR 2018 - Vancouver, Canada
Duration: Apr 30 2018May 3 2018

Conference

Conference6th International Conference on Learning Representations, ICLR 2018
CountryCanada
CityVancouver
Period4/30/185/3/18

    Fingerprint

ASJC Scopus subject areas

  • Education
  • Computer Science Applications
  • Linguistics and Language
  • Language and Linguistics

Cite this

Liang, S., Sun, R., Srikant, R., & Li, Y. (2018). Understanding the loss surface of single-layered neural networks for binary classification. Paper presented at 6th International Conference on Learning Representations, ICLR 2018, Vancouver, Canada.