Asynchronous Methods for Deep Reinforcement Learning

Abstract

We propose a conceptually simple and lightweight framework for deep reinforcement learning that uses asynchronous gradient descent for optimization of deep neural network controllers. We present asynchronous variants of four standard reinforcement learning algorithms and show that parallel actor-learners have a stabilizing effect on training allowing all four methods to successfully train neural network controllers. The best performing method, an asynchronous variant of actor-critic, surpasses the current state-of-the-art on the Atari domain while training for half the time on a single multi-core CPU instead of a GPU. Furthermore, we show that asynchronous actor-critic succeeds on a wide variety of continuous motor control problems as well as on a new task of navigating random 3D mazes using a visual input.

References in corpus (2)

• ADADELTA: An Adaptive Learning Rate Method
• Massively Parallel Methods for Deep Reinforcement Learning

Cited by in corpus (126)

• A Survey and Critique of Multiagent Deep Reinforcement Learning
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Unifying Count-Based Exploration and Intrinsic Motivation
• Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning
• First return, then explore
• Federated Reinforcement Learning: Techniques, Applications, and Open Challenges
• Sample Efficient Actor-Critic with Experience Replay
• Quantum agents in the Gym: a variational quantum algorithm for deep Q-learning
• dm_control: Software and Tasks for Continuous Control
• Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?
• Sim4CV: A Photo-Realistic Simulator for Computer Vision Applications
• Equivalence Between Policy Gradients and Soft Q-Learning
• Recent advances in applying deep reinforcement learning for flow control: perspectives and future directions
• Third-Person Imitation Learning
• A Deep Hierarchical Approach to Lifelong Learning in Minecraft
• When does reinforcement learning stand out in quantum control? A comparative study on state preparation
• DARLA: Improving Zero-Shot Transfer in Reinforcement Learning
• Generative Design by Reinforcement Learning: Enhancing the Diversity of Topology Optimization Designs
• Deep Learning and Knowledge-Based Methods for Computer Aided Molecular Design -- Toward a Unified Approach: State-of-the-Art and Future Directions
• A review on Deep Reinforcement Learning for Fluid Mechanics
• A review on deep reinforcement learning for fluid mechanics: an update
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Loss is its own Reward: Self-Supervision for Reinforcement Learning
• Safe and Efficient Off-Policy Reinforcement Learning
• Combining policy gradient and Q-learning
• Deep Successor Reinforcement Learning
• Explaining and Interpreting LSTMs
• DRLinFluids -- An open-source python platform of coupling Deep Reinforcement Learning and OpenFOAM
• A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation
• Some Considerations on Learning to Explore via Meta-Reinforcement Learning
• Combining Deep Reinforcement Learning and Safety Based Control for Autonomous Driving
• Learning and Querying Fast Generative Models for Reinforcement Learning
• ELF: An Extensive, Lightweight and Flexible Research Platform for Real-time Strategy Games
• Towards better decoding and language model integration in sequence to sequence models
• Learning to Perform Physics Experiments via Deep Reinforcement Learning
• Learning What Data to Learn
• Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges
• Socially Aware Motion Planning with Deep Reinforcement Learning
• Distributed Deep Reinforcement Learning: A Survey and A Multi-Player Multi-Agent Learning Toolbox
• The Two Kinds of Free Energy and the Bayesian Revolution
• A Survey of Deep Learning for Data Caching in Edge Network
• V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control
• Warmth and competence in human-agent cooperation
• Towards a Common Implementation of Reinforcement Learning for Multiple Robotic Tasks
• Local minima in training of neural networks
• LIFT: Reinforcement Learning in Computer Systems by Learning From Demonstrations
• Utilizing Reinforcement Learning for de novo Drug Design
• Reinforcement Learning for Robotic Manipulation using Simulated Locomotion Demonstrations
• AI in Human-computer Gaming: Techniques, Challenges and Opportunities
• Robust Multimodal Image Registration Using Deep Recurrent Reinforcement Learning
• TD-Regularized Actor-Critic Methods
• Using Reinforcement Learning in the Algorithmic Trading Problem
• Multi-focus Attention Network for Efficient Deep Reinforcement Learning
• Classification with Costly Features as a Sequential Decision-Making Problem
• Multi-Agent Reinforcement Learning for Dynamic Ocean Monitoring by a Swarm of Buoys
• Option Discovery in Hierarchical Reinforcement Learning using Spatio-Temporal Clustering
• Deep Controlled Learning for Inventory Control
• Neural Program Synthesis with Priority Queue Training
• Model-Free Design of Control Systems over Wireless Fading Channels
• Pre-training with Non-expert Human Demonstration for Deep Reinforcement Learning
• Safe Option-Critic: Learning Safety in the Option-Critic Architecture
• Learning Reciprocity in Complex Sequential Social Dilemmas
• Rethinking Closed-loop Training for Autonomous Driving
• Cyber-Physical Defense in the Quantum Era
• Quality of service based radar resource management using deep reinforcement learning
• Generalization in Transfer Learning
• Learning to Manipulate Deformable Objects without Demonstrations
• Collaborative Deep Reinforcement Learning
• An Introduction to Deep Learning for the Physical Layer
• Training Agents using Upside-Down Reinforcement Learning
• DRLDO: A novel DRL based De-ObfuscationSystem for Defense against Metamorphic Malware
• Weakly Supervised Reinforcement Learning for Autonomous Highway Driving via Virtual Safety Cages
• Deep Reinforcement Learning for Contact-Rich Skills Using Compliant Movement Primitives
• Routing algorithms as tools for integrating social distancing with emergency evacuation
• Probabilistic design of optimal sequential decision-making algorithms in learning and control
• Parallel Exploration via Negatively Correlated Search
• Tracking-by-Trackers with a Distilled and Reinforced Model
• Reinforcement Learning for Systematic FX Trading
• On Wasserstein Reinforcement Learning and the Fokker-Planck equation
• ESceme: Vision-and-Language Navigation with Episodic Scene Memory
• Deep Reinforcement Learning With Macro-Actions
• Learning Adaptive Exploration Strategies in Dynamic Environments Through Informed Policy Regularization
• Improving Model-Based Reinforcement Learning with Internal State Representations through Self-Supervision
• Control of nonlinear, complex and black-boxed greenhouse system with reinforcement learning
• A review of motion planning algorithms for intelligent robotics
• Entanglement engineering of optomechanical systems by reinforcement learning
• Entropy-Aware Model Initialization for Effective Exploration in Deep Reinforcement Learning
• Combining Subgoal Graphs with Reinforcement Learning to Build a Rational Pathfinder
• Mutual influence between language and perception in multi-agent communication games
• Real-time visual tracking by deep reinforced decision making
• Comparing Deep Reinforcement Learning Algorithms in Two-Echelon Supply Chains
• Flow-based Intrinsic Curiosity Module
• Learning to Communicate Using Counterfactual Reasoning
• Investigating Recurrence and Eligibility Traces in Deep Q-Networks
• Analysing Factorizations of Action-Value Networks for Cooperative Multi-Agent Reinforcement Learning
• Design of Restricted Normalizing Flow towards Arbitrary Stochastic Policy with Computational Efficiency
• Feasibility-Guided Learning for Robust Control in Constrained Optimal Control Problems
• Boosting Deep Reinforcement Learning with Semantic Knowledge for Robotic Manipulators
• Compressed Federated Reinforcement Learning with a Generative Model
• Globally Optimal Hierarchical Reinforcement Learning for Linearly-Solvable Markov Decision Processes
• Learning Invariances for Policy Generalization
• A short variational proof of equivalence between policy gradients and soft Q learning
• Group-Agent Reinforcement Learning
• Optimal preference satisfaction for conflict-free joint decisions
• Fully Distributed Actor-Critic Architecture for Multitask Deep Reinforcement Learning
• DeepNav: Learning to Navigate Large Cities
• Capsule Network Performance with Autonomous Navigation
• Multi-agent Path Finding for Timed Tasks using Evolutionary Games
• Classification with Costly Features in Hierarchical Deep Sets
• Learning-based Hamilton-Jacobi-Bellman Methods for Optimal Control
• Sample Efficient Reinforcement Learning via Large Vision Language Model Distillation
• Distributed Multi-Agent Deep Reinforcement Learning Framework for Whole-building HVAC Control
• Tuning Synaptic Connections instead of Weights by Genetic Algorithm in Spiking Policy Network
• MDP Playground: An Analysis and Debug Testbed for Reinforcement Learning
• An advantage actor-critic algorithm for robotic motion planning in dense and dynamic scenarios
• A survey of benchmarking frameworks for reinforcement learning
• Deep RL Agent for a Real-Time Action Strategy Game
• FORLORN: A Framework for Comparing Offline Methods and Reinforcement Learning for Optimization of RAN Parameters
• Biological Blueprints for Next Generation AI Systems
• Conditioning of Reinforcement Learning Agents and its Policy Regularization Application
• Policy-Based Reinforcement Learning for Assortative Matching in Human Behavior Modeling
• Reinforcement Learning for Volt-Var Control: A Novel Two-stage Progressive Training Strategy
• Solving Atari Games Using Fractals And Entropy
• Using reinforcement learning to design an AI assistantfor a satisfying co-op experience
• A Threshold-based Scheme for Reinforcement Learning in Neural Networks