Diff-DAC: Distributed Actor-Critic for Average Multitask Deep Reinforcement Learning
arXiv:1710.10363
Abstract
We propose a fully distributed actor-critic algorithm approximated by deep neural networks, named \textit{Diff-DAC}, with application to single-task and to average multitask reinforcement learning (MRL). Each agent has access to data from its local task only, but it aims to learn a policy that performs well on average for the whole set of tasks. During the learning process, agents communicate their value-policy parameters to their neighbors, diffusing the information across the network, so that they converge to a common policy, with no need for a central node. The method is scalable, since the computational and communication costs per agent grow with its number of neighbors. We derive Diff-DAC's from duality theory and provide novel insights into the standard actor-critic framework, showing that it is actually an instance of the dual ascent method that approximates the solution of a linear program. Experiments suggest that Diff-DAC can outperform the single previous distributed MRL approach (i.e., Dist-MTLPS) and even the centralized architecture.
References in corpus (12)
- Continuous control with deep reinforcement learning
- Playing Atari with Deep Reinforcement Learning
- How transferable are features in deep neural networks?
- Gaussian Processes for Data-Efficient Learning in Robotics and Control
- Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
- Modular Multitask Reinforcement Learning with Policy Sketches
- Distributed Pareto Optimization via Diffusion Strategies
- Distral: Robust Multitask Reinforcement Learning
- -Learning: A Collaborative Distributed Strategy for Multi-Agent Reinforcement Learning Through Consensus + Innovations
- Performance Limits for Distributed Estimation Over LMS Adaptive Networks
- Connecting Generative Adversarial Networks and Actor-Critic Methods
- A Framework for Parallel and Distributed Training of Neural Networks
Cited by in corpus (5)
- Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents
- Multi-Agent Reinforcement Learning via Double Averaging Primal-Dual Optimization
- RODE: Learning Roles to Decompose Multi-Agent Tasks
- Off-Policy Multi-Agent Decomposed Policy Gradients
- How to Organize your Deep Reinforcement Learning Agents: The Importance of Communication Topology