Stochastic Learning Rate With Memory: Optimization in the Stochastic Approximation and Online Learning Settings

Theodoros Mamalis, Dusan Stipanovic, Petros Voulgaris

Research output: Contribution to journalArticlepeer-review

Abstract

In this letter, multiplicative stochasticity is applied to the learning rate of stochastic optimization algorithms, giving rise to stochastic learning-rate schemes. In-expectation theoretical convergence results of Stochastic Gradient Descent equipped with this novel learning rate scheme under the stochastic setting, as well as convergence results under the online optimization settings are provided. Empirical results consider the case of an adaptively uniformly distributed multiplicative stochasticity and include not only Stochastic Gradient Descent, but also other popular algorithms equipped with a stochastic learning rate. They demonstrate noticeable optimization performance gains with respect to their deterministic-learning-rate versions.

Original languageEnglish (US)
Pages (from-to)419-424
Number of pages6
JournalIEEE Control Systems Letters
Volume7
DOIs
StatePublished - 2023

Keywords

  • Optimization algorithms
  • convergence
  • machine learning
  • stochastic systems

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Control and Optimization

Fingerprint

Dive into the research topics of 'Stochastic Learning Rate With Memory: Optimization in the Stochastic Approximation and Online Learning Settings'. Together they form a unique fingerprint.

Cite this