Decentralized Federated Learning for Over-Parameterized Models

Tiancheng Qin; S. Rasoul Etesami; Cesar A. Uribe

doi:10.1109/CDC51059.2022.9992924

Decentralized Federated Learning for Over-Parameterized Models

Tiancheng Qin, S. Rasoul Etesami, Cesar A. Uribe

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Modern machine learning, especially deep learning, features models that are often highly expressive and over-parameterized. They can interpolate the data by driving the empirical loss close to zero. We analyze the convergence rate of decentralized stochastic gradient descent (SGD), which is at the core of decentralized federated learning (DFL), for these over-parameterized models. Our analysis covers the setting of decentralized SGD with time-varying networks, local updates and heterogeneous data. We establish strong convergence guarantees with or without the assumption of convex objectives that either improves upon the existing literature or is the first for the regime.

Original language	English (US)
Title of host publication	2022 IEEE 61st Conference on Decision and Control, CDC 2022
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	5200-5205
Number of pages	6
ISBN (Electronic)	9781665467612
DOIs	https://doi.org/10.1109/CDC51059.2022.9992924
State	Published - 2022
Externally published	Yes
Event	61st IEEE Conference on Decision and Control, CDC 2022 - Cancun, Mexico Duration: Dec 6 2022 → Dec 9 2022

Publication series

Name	Proceedings of the IEEE Conference on Decision and Control
Volume	2022-December
ISSN (Print)	0743-1546
ISSN (Electronic)	2576-2370

Conference

Conference	61st IEEE Conference on Decision and Control, CDC 2022
Country/Territory	Mexico
City	Cancun
Period	12/6/22 → 12/9/22

Keywords

Decentralized Federated Learning
Decentralized Optimization
Local SGD
Overparameterization

ASJC Scopus subject areas

Control and Systems Engineering
Modeling and Simulation
Control and Optimization

Online availability

10.1109/CDC51059.2022.9992924

Library availability

Discover UIUC Full Text

Cite this

Qin, T., Etesami, S. R., & Uribe, C. A. (2022). Decentralized Federated Learning for Over-Parameterized Models. In 2022 IEEE 61st Conference on Decision and Control, CDC 2022 (pp. 5200-5205). (Proceedings of the IEEE Conference on Decision and Control; Vol. 2022-December). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/CDC51059.2022.9992924

Decentralized Federated Learning for Over-Parameterized Models. / Qin, Tiancheng; Etesami, S. Rasoul; Uribe, Cesar A.
2022 IEEE 61st Conference on Decision and Control, CDC 2022. Institute of Electrical and Electronics Engineers Inc., 2022. p. 5200-5205 (Proceedings of the IEEE Conference on Decision and Control; Vol. 2022-December).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Qin, T, Etesami, SR & Uribe, CA 2022, Decentralized Federated Learning for Over-Parameterized Models. in 2022 IEEE 61st Conference on Decision and Control, CDC 2022. Proceedings of the IEEE Conference on Decision and Control, vol. 2022-December, Institute of Electrical and Electronics Engineers Inc., pp. 5200-5205, 61st IEEE Conference on Decision and Control, CDC 2022, Cancun, Mexico, 12/6/22. https://doi.org/10.1109/CDC51059.2022.9992924

@inproceedings{f26810dc4a354489b4225a99c2c2142e,

title = "Decentralized Federated Learning for Over-Parameterized Models",

abstract = "Modern machine learning, especially deep learning, features models that are often highly expressive and over-parameterized. They can interpolate the data by driving the empirical loss close to zero. We analyze the convergence rate of decentralized stochastic gradient descent (SGD), which is at the core of decentralized federated learning (DFL), for these over-parameterized models. Our analysis covers the setting of decentralized SGD with time-varying networks, local updates and heterogeneous data. We establish strong convergence guarantees with or without the assumption of convex objectives that either improves upon the existing literature or is the first for the regime.",

keywords = "Decentralized Federated Learning, Decentralized Optimization, Local SGD, Overparameterization",

author = "Tiancheng Qin and Etesami, {S. Rasoul} and Uribe, {Cesar A.}",

note = "This material is based upon work supported by the National Science Foundation under Grants No. EPCN-1944403, No. 2211815, and No. 2213568.; 61st IEEE Conference on Decision and Control, CDC 2022 ; Conference date: 06-12-2022 Through 09-12-2022",

year = "2022",

doi = "10.1109/CDC51059.2022.9992924",

language = "English (US)",

series = "Proceedings of the IEEE Conference on Decision and Control",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "5200--5205",

booktitle = "2022 IEEE 61st Conference on Decision and Control, CDC 2022",

address = "United States",

}

TY - GEN

T1 - Decentralized Federated Learning for Over-Parameterized Models

AU - Qin, Tiancheng

AU - Etesami, S. Rasoul

AU - Uribe, Cesar A.

N1 - This material is based upon work supported by the National Science Foundation under Grants No. EPCN-1944403, No. 2211815, and No. 2213568.

PY - 2022

Y1 - 2022

N2 - Modern machine learning, especially deep learning, features models that are often highly expressive and over-parameterized. They can interpolate the data by driving the empirical loss close to zero. We analyze the convergence rate of decentralized stochastic gradient descent (SGD), which is at the core of decentralized federated learning (DFL), for these over-parameterized models. Our analysis covers the setting of decentralized SGD with time-varying networks, local updates and heterogeneous data. We establish strong convergence guarantees with or without the assumption of convex objectives that either improves upon the existing literature or is the first for the regime.

AB - Modern machine learning, especially deep learning, features models that are often highly expressive and over-parameterized. They can interpolate the data by driving the empirical loss close to zero. We analyze the convergence rate of decentralized stochastic gradient descent (SGD), which is at the core of decentralized federated learning (DFL), for these over-parameterized models. Our analysis covers the setting of decentralized SGD with time-varying networks, local updates and heterogeneous data. We establish strong convergence guarantees with or without the assumption of convex objectives that either improves upon the existing literature or is the first for the regime.

KW - Decentralized Federated Learning

KW - Decentralized Optimization

KW - Local SGD

KW - Overparameterization

UR - http://www.scopus.com/inward/record.url?scp=85147013639&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85147013639&partnerID=8YFLogxK

U2 - 10.1109/CDC51059.2022.9992924

DO - 10.1109/CDC51059.2022.9992924

M3 - Conference contribution

AN - SCOPUS:85147013639

T3 - Proceedings of the IEEE Conference on Decision and Control

SP - 5200

EP - 5205

BT - 2022 IEEE 61st Conference on Decision and Control, CDC 2022

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 61st IEEE Conference on Decision and Control, CDC 2022

Y2 - 6 December 2022 through 9 December 2022

ER -

Decentralized Federated Learning for Over-Parameterized Models

Abstract

Publication series

Conference

Keywords

ASJC Scopus subject areas

Online availability

Library availability

Related links

Fingerprint

Cite this