Efficient Parallel Methods for Deep Reinforcement Learning

Abstract

We propose a novel framework for efficient parallelization of deep reinforcement learning algorithms, enabling these algorithms to learn from multiple actors on a single machine. The framework is algorithm agnostic and can be applied to on-policy, off-policy, value based and policy gradient based algorithms. Given its inherent parallelism, the framework can be efficiently implemented on a GPU, allowing the usage of powerful models while significantly reducing training time. We demonstrate the effectiveness of our framework by implementing an advantage actor-critic algorithm on a GPU, using on-policy experiences and employing synchronous updates. Our algorithm achieves state-of-the-art performance on the Atari domain after only a few hours of training. Our framework thus opens the door for much faster experimentation on demanding problem domains. Our implementation is open-source and is made public at https://github.com/alfredvc/paac

References in corpus (2)

• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Massively Parallel Methods for Deep Reinforcement Learning

Cited by in corpus (22)

• Distributed Prioritized Experience Replay
• A Survey of Beam Management for mmWave and THz Communications Towards 6G
• Federated Reinforcement Learning: Techniques, Applications, and Open Challenges
• Autonomous Unmanned Aerial Vehicle Navigation using Reinforcement Learning: A Systematic Review
• Tianshou: a Highly Modularized Deep Reinforcement Learning Library
• Accelerated Methods for Deep Reinforcement Learning
• A Survey on Vulnerability of Federated Learning: A Learning Algorithm Perspective
• TrojDRL: Trojan Attacks on Deep Reinforcement Learning Agents
• Hierarchical Reinforcement Learning By Discovering Intrinsic Options
• Gossip-based Actor-Learner Architectures for Deep Reinforcement Learning
• Guided Feature Transformation (GFT): A Neural Language Grounding Module for Embodied Agents
• Accelerating Reinforcement Learning through GPU Atari Emulation
• TLeague: A Framework for Competitive Self-Play based Distributed Multi-Agent Reinforcement Learning
• Population-Guided Parallel Policy Search for Reinforcement Learning
• Generalized Off-Policy Actor-Critic
• Provably Efficient Cooperative Multi-Agent Reinforcement Learning with Function Approximation
• Continual and Multi-task Reinforcement Learning With Shared Episodic Memory
• Distributed Deep Reinforcement Learning: An Overview
• Fast Retinomorphic Event Stream for Video Recognition and Reinforcement Learning
• Parallel Actors and Learners: A Framework for Generating Scalable RL Implementations
• Human-Level Control without Server-Grade Hardware
• Communication Efficient Parallel Reinforcement Learning