Faster Directional Convergence of Linear Neural Networks under Spherically Symmetric Data

Dachao Lin; Ruoyu Sun; Zhihua Zhang

Faster Directional Convergence of Linear Neural Networks under Spherically Symmetric Data

Dachao Lin, Ruoyu Sun, Zhihua Zhang

Industrial and Enterprise Systems Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

In this paper, we study gradient methods for training deep linear neural networks with binary cross-entropy loss. In particular, we show global directional convergence guarantees from a polynomial rate to a linear rate for (deep) linear networks with spherically symmetric data distribution, which can be viewed as a specific zero-margin dataset. Our results do not require the assumptions in other works such as small initial loss, presumed convergence of weight direction, or overparameterization. We also characterize our findings in experiments.

Original language	English (US)
Title of host publication	Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021
Editors	Marc'Aurelio Ranzato, Alina Beygelzimer, Yann Dauphin, Percy S. Liang, Jenn Wortman Vaughan
Publisher	Neural information processing systems foundation
Pages	4647-4660
Number of pages	14
ISBN (Electronic)	9781713845393
State	Published - 2021
Event	35th Conference on Neural Information Processing Systems, NeurIPS 2021 - Virtual, Online Duration: Dec 6 2021 → Dec 14 2021

Publication series

Name	Advances in Neural Information Processing Systems
Volume	6
ISSN (Print)	1049-5258

Conference

Conference	35th Conference on Neural Information Processing Systems, NeurIPS 2021
City	Virtual, Online
Period	12/6/21 → 12/14/21

ASJC Scopus subject areas

Computer Networks and Communications
Information Systems
Signal Processing

Library availability

Discover UIUC Full Text

Cite this

Lin, D., Sun, R., & Zhang, Z. (2021). Faster Directional Convergence of Linear Neural Networks under Spherically Symmetric Data. In MA. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, & J. Wortman Vaughan (Eds.), Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021 (pp. 4647-4660). (Advances in Neural Information Processing Systems; Vol. 6). Neural information processing systems foundation.

Faster Directional Convergence of Linear Neural Networks under Spherically Symmetric Data. / Lin, Dachao; Sun, Ruoyu; Zhang, Zhihua.
Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021. ed. / Marc'Aurelio Ranzato; Alina Beygelzimer; Yann Dauphin; Percy S. Liang; Jenn Wortman Vaughan. Neural information processing systems foundation, 2021. p. 4647-4660 (Advances in Neural Information Processing Systems; Vol. 6).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Lin, D, Sun, R & Zhang, Z 2021, Faster Directional Convergence of Linear Neural Networks under Spherically Symmetric Data. in MA Ranzato, A Beygelzimer, Y Dauphin, PS Liang & J Wortman Vaughan (eds), Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021. Advances in Neural Information Processing Systems, vol. 6, Neural information processing systems foundation, pp. 4647-4660, 35th Conference on Neural Information Processing Systems, NeurIPS 2021, Virtual, Online, 12/6/21.

Lin D, Sun R, Zhang Z. Faster Directional Convergence of Linear Neural Networks under Spherically Symmetric Data. In Ranzato MA, Beygelzimer A, Dauphin Y, Liang PS, Wortman Vaughan J, editors, Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021. Neural information processing systems foundation. 2021. p. 4647-4660. (Advances in Neural Information Processing Systems).

Lin, Dachao ; Sun, Ruoyu ; Zhang, Zhihua. / Faster Directional Convergence of Linear Neural Networks under Spherically Symmetric Data. Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021. editor / Marc'Aurelio Ranzato ; Alina Beygelzimer ; Yann Dauphin ; Percy S. Liang ; Jenn Wortman Vaughan. Neural information processing systems foundation, 2021. pp. 4647-4660 (Advances in Neural Information Processing Systems).

@inproceedings{54d292d406f14bfabf1d48fac85b08ec,

title = "Faster Directional Convergence of Linear Neural Networks under Spherically Symmetric Data",

abstract = "In this paper, we study gradient methods for training deep linear neural networks with binary cross-entropy loss. In particular, we show global directional convergence guarantees from a polynomial rate to a linear rate for (deep) linear networks with spherically symmetric data distribution, which can be viewed as a specific zero-margin dataset. Our results do not require the assumptions in other works such as small initial loss, presumed convergence of weight direction, or overparameterization. We also characterize our findings in experiments.",

author = "Dachao Lin and Ruoyu Sun and Zhihua Zhang",

note = "Publisher Copyright: {\textcopyright} 2021 Neural information processing systems foundation. All rights reserved.; 35th Conference on Neural Information Processing Systems, NeurIPS 2021 ; Conference date: 06-12-2021 Through 14-12-2021",

year = "2021",

language = "English (US)",

series = "Advances in Neural Information Processing Systems",

publisher = "Neural information processing systems foundation",

pages = "4647--4660",

editor = "Marc'Aurelio Ranzato and Alina Beygelzimer and Yann Dauphin and Liang, {Percy S.} and {Wortman Vaughan}, Jenn",

booktitle = "Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021",

}

TY - GEN

T1 - Faster Directional Convergence of Linear Neural Networks under Spherically Symmetric Data

AU - Lin, Dachao

AU - Sun, Ruoyu

AU - Zhang, Zhihua

PY - 2021

Y1 - 2021

N2 - In this paper, we study gradient methods for training deep linear neural networks with binary cross-entropy loss. In particular, we show global directional convergence guarantees from a polynomial rate to a linear rate for (deep) linear networks with spherically symmetric data distribution, which can be viewed as a specific zero-margin dataset. Our results do not require the assumptions in other works such as small initial loss, presumed convergence of weight direction, or overparameterization. We also characterize our findings in experiments.

AB - In this paper, we study gradient methods for training deep linear neural networks with binary cross-entropy loss. In particular, we show global directional convergence guarantees from a polynomial rate to a linear rate for (deep) linear networks with spherically symmetric data distribution, which can be viewed as a specific zero-margin dataset. Our results do not require the assumptions in other works such as small initial loss, presumed convergence of weight direction, or overparameterization. We also characterize our findings in experiments.

UR - http://www.scopus.com/inward/record.url?scp=85131745435&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85131745435&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85131745435

T3 - Advances in Neural Information Processing Systems

SP - 4647

EP - 4660

BT - Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021

A2 - Ranzato, Marc'Aurelio

A2 - Beygelzimer, Alina

A2 - Dauphin, Yann

A2 - Liang, Percy S.

A2 - Wortman Vaughan, Jenn

PB - Neural information processing systems foundation

T2 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021

Y2 - 6 December 2021 through 14 December 2021

ER -

Faster Directional Convergence of Linear Neural Networks under Spherically Symmetric Data

Abstract

Publication series

Conference

ASJC Scopus subject areas

Library availability

Related links

Fingerprint

Cite this