Communication-Efficient Policy Gradient Methods for Distributed Reinforcement Learning

Tianyi Chen, Kaiqing Zhang, Georgios B. Giannakis, Tamer Basar

Research output: Contribution to journalArticlepeer-review

Abstract

This article deals with distributed policy optimization in reinforcement learning, which involves a central controller and a group of learners. In particular, two typical settings encountered in several applications are considered: multiagent reinforcement learning (RL) and parallel RL, where frequent information exchanges between the learners and the controller are required. For many practical distributed systems, however, the overhead caused by these frequent communication exchanges is considerable, and becomes the bottleneck of the overall performance. To address this challenge, a novel policy gradient approach is developed for solving distributed RL. The novel approach adaptively skips the policy gradient communication during iterations, and can reduce the communication overhead without degrading learning performance. It is established analytically that: i) the novel algorithm has a convergence rate identical to that of the plain-vanilla policy gradient; while ii) if the distributed learners are heterogeneous in terms of their reward functions, the number of communication rounds needed to achieve a desirable learning accuracy is markedly reduced. Numerical experiments corroborate the communication reduction attained by the novel algorithm compared to alternatives.

Original languageEnglish (US)
Pages (from-to)917-929
Number of pages13
JournalIEEE Transactions on Control of Network Systems
Volume9
Issue number2
DOIs
StatePublished - Jun 1 2022
Externally publishedYes

Keywords

  • Aerospace electronics
  • Approximation algorithms
  • Collaboration
  • Convergence
  • Reinforcement learning
  • Task analysis
  • Trajectory
  • communication-efficient learning
  • distributed learning
  • multi-agent
  • policy gradient
  • reinforcement learning
  • Communication-efficient learning
  • multiagent

ASJC Scopus subject areas

  • Control and Optimization
  • Signal Processing
  • Control and Systems Engineering
  • Computer Networks and Communications

Fingerprint

Dive into the research topics of 'Communication-Efficient Policy Gradient Methods for Distributed Reinforcement Learning'. Together they form a unique fingerprint.

Cite this