Communication Efficient Parallel Reinforcement Learning

Abstract

We consider the problem where $M$ agents interact with $M$ identical and independent environments with $S$ states and $A$ actions using reinforcement learning for $T$ rounds. The agents share their data with a central server to minimize their regret. We aim to find an algorithm that allows the agents to minimize the regret with infrequent communication rounds. We provide \NAM\ which runs at each agent and prove that the total cumulative regret of $M$ agents is upper bounded as $\Tilde{O}(DS\sqrt{MAT})$ for a Markov Decision Process with diameter $D$, number of states $S$, and number of actions $A$. The agents synchronize after their visitations to any state-action pair exceeds a certain threshold. Using this, we obtain a bound of $O\left(MSA\log (MT)\right)$ on the total number of communications rounds. Finally, we evaluate the algorithm against multiple environments and demonstrate that the proposed algorithm performs at par with an always communication version of the UCRL2 algorithm, while with significantly lower communication.

References in corpus (10)

• Playing Atari with Deep Reinforcement Learning
• Trust Region Policy Optimization
• Massively Parallel Methods for Deep Reinforcement Learning
• (More) Efficient Reinforcement Learning via Posterior Sampling
• DeepPool: Distributed Model-free Algorithm for Ride-sharing using Deep Reinforcement Learning
• REGAL: A Regularization based Algorithm for Reinforcement Learning in Weakly Communicating MDPs
• Efficient Parallel Methods for Deep Reinforcement Learning
• Distributed Exploration in Multi-Armed Bandits
• Distributed Non-Stochastic Experts
• Near-optimal Optimistic Reinforcement Learning using Empirical Bernstein Inequalities