Communication-Efficient Policy Gradient Methods for Distributed Reinforcement Learning

Abstract

This paper deals with distributed policy optimization in reinforcement learning, which involves a central controller and a group of learners. In particular, two typical settings encountered in several applications are considered: multi-agent reinforcement learning (RL) and parallel RL, where frequent information exchanges between the learners and the controller are required. For many practical distributed systems, however, the overhead caused by these frequent communication exchanges is considerable, and becomes the bottleneck of the overall performance. To address this challenge, a novel policy gradient approach is developed for solving distributed RL. The novel approach adaptively skips the policy gradient communication during iterations, and can reduce the communication overhead without degrading learning performance. It is established analytically that: i) the novel algorithm has convergence rate identical to that of the plain-vanilla policy gradient; while ii) if the distributed learners are heterogeneous in terms of their reward functions, the number of communication rounds needed to achieve a desirable learning accuracy is markedly reduced. Numerical experiments corroborate the communication reduction attained by the novel algorithm compared to alternatives.

arXiv commentTo appear in IEEE Transactions on Control of Network Systems

References in corpus (13)

• Continuous control with deep reinforcement learning
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Trust Region Policy Optimization
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• Massively Parallel Methods for Deep Reinforcement Learning
• Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents
• Local SGD Converges Fast and Communicates Little
• $QD$-Learning: A Collaborative Distributed Strategy for Multi-Agent Reinforcement Learning Through Consensus + Innovations
• Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability
• A Berkeley View of Systems Challenges for AI
• Bandit Convex Optimization for Scalable and Dynamic IoT Management
• Multi-Agent Reinforcement Learning via Double Averaging Primal-Dual Optimization