Asynchronous Methods for Deep Reinforcement Learning
arXiv:1602.01783
Abstract
We propose a conceptually simple and lightweight framework for deep reinforcement learning that uses asynchronous gradient descent for optimization of deep neural network controllers. We present asynchronous variants of four standard reinforcement learning algorithms and show that parallel actor-learners have a stabilizing effect on training allowing all four methods to successfully train neural network controllers. The best performing method, an asynchronous variant of actor-critic, surpasses the current state-of-the-art on the Atari domain while training for half the time on a single multi-core CPU instead of a GPU. Furthermore, we show that asynchronous actor-critic succeeds on a wide variety of continuous motor control problems as well as on a new task of navigating random 3D mazes using a visual input.
References in corpus (10)
- Adam: A Method for Stochastic Optimization
- Continuous control with deep reinforcement learning
- ADADELTA: An Adaptive Learning Rate Method
- Playing Atari with Deep Reinforcement Learning
- Prioritized Experience Replay
- High-Dimensional Continuous Control Using Generalized Advantage Estimation
- End-to-End Training of Deep Visuomotor Policies
- HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
- Massively Parallel Methods for Deep Reinforcement Learning
- Distributed Deep Q-Learning
Cited by in corpus (127)
- A Survey and Critique of Multiagent Deep Reinforcement Learning
- PathNet: Evolution Channels Gradient Descent in Super Neural Networks
- RL: Fast Reinforcement Learning via Slow Reinforcement Learning
- Unifying Count-Based Exploration and Intrinsic Motivation
- Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning
- First return, then explore
- Federated Reinforcement Learning: Techniques, Applications, and Open Challenges
- Sample Efficient Actor-Critic with Experience Replay
- Quantum agents in the Gym: a variational quantum algorithm for deep Q-learning
- dm_control: Software and Tasks for Continuous Control
- Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?
- Sim4CV: A Photo-Realistic Simulator for Computer Vision Applications
- Equivalence Between Policy Gradients and Soft Q-Learning
- Recent advances in applying deep reinforcement learning for flow control: perspectives and future directions
- Third-Person Imitation Learning
- A Deep Hierarchical Approach to Lifelong Learning in Minecraft
- When does reinforcement learning stand out in quantum control? A comparative study on state preparation
- DARLA: Improving Zero-Shot Transfer in Reinforcement Learning
- Generative Design by Reinforcement Learning: Enhancing the Diversity of Topology Optimization Designs
- Deep Learning and Knowledge-Based Methods for Computer Aided Molecular Design -- Toward a Unified Approach: State-of-the-Art and Future Directions
- A review on deep reinforcement learning for fluid mechanics: an update
- A review on Deep Reinforcement Learning for Fluid Mechanics
- Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
- Safe and Efficient Off-Policy Reinforcement Learning
- Loss is its own Reward: Self-Supervision for Reinforcement Learning
- Combining policy gradient and Q-learning
- Deep Successor Reinforcement Learning
- Explaining and Interpreting LSTMs
- DRLinFluids -- An open-source python platform of coupling Deep Reinforcement Learning and OpenFOAM
- A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation
- Some Considerations on Learning to Explore via Meta-Reinforcement Learning
- Combining Deep Reinforcement Learning and Safety Based Control for Autonomous Driving
- Learning and Querying Fast Generative Models for Reinforcement Learning
- ELF: An Extensive, Lightweight and Flexible Research Platform for Real-time Strategy Games
- Towards better decoding and language model integration in sequence to sequence models
- Learning to Perform Physics Experiments via Deep Reinforcement Learning
- Learning What Data to Learn
- Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges
- The Two Kinds of Free Energy and the Bayesian Revolution
- Distributed Deep Reinforcement Learning: A Survey and A Multi-Player Multi-Agent Learning Toolbox
- Socially Aware Motion Planning with Deep Reinforcement Learning
- A Survey of Deep Learning for Data Caching in Edge Network
- V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control
- Warmth and competence in human-agent cooperation
- Local minima in training of neural networks
- Towards a Common Implementation of Reinforcement Learning for Multiple Robotic Tasks
- Utilizing Reinforcement Learning for de novo Drug Design
- LIFT: Reinforcement Learning in Computer Systems by Learning From Demonstrations
- AI in Human-computer Gaming: Techniques, Challenges and Opportunities
- Reinforcement Learning for Robotic Manipulation using Simulated Locomotion Demonstrations
- Robust Multimodal Image Registration Using Deep Recurrent Reinforcement Learning
- TD-Regularized Actor-Critic Methods
- Multi-focus Attention Network for Efficient Deep Reinforcement Learning
- Using Reinforcement Learning in the Algorithmic Trading Problem
- Classification with Costly Features as a Sequential Decision-Making Problem
- Multi-Agent Reinforcement Learning for Dynamic Ocean Monitoring by a Swarm of Buoys
- Option Discovery in Hierarchical Reinforcement Learning using Spatio-Temporal Clustering
- Neural Program Synthesis with Priority Queue Training
- Deep Controlled Learning for Inventory Control
- Model-Free Design of Control Systems over Wireless Fading Channels
- Safe Option-Critic: Learning Safety in the Option-Critic Architecture
- Pre-training with Non-expert Human Demonstration for Deep Reinforcement Learning
- Learning Reciprocity in Complex Sequential Social Dilemmas
- Rethinking Closed-loop Training for Autonomous Driving
- Cyber-Physical Defense in the Quantum Era
- Quality of service based radar resource management using deep reinforcement learning
- Generalization in Transfer Learning
- An Introduction to Deep Learning for the Physical Layer
- Collaborative Deep Reinforcement Learning
- Learning to Manipulate Deformable Objects without Demonstrations
- Training Agents using Upside-Down Reinforcement Learning
- DRLDO: A novel DRL based De-ObfuscationSystem for Defense against Metamorphic Malware
- Weakly Supervised Reinforcement Learning for Autonomous Highway Driving via Virtual Safety Cages
- Deep Reinforcement Learning for Contact-Rich Skills Using Compliant Movement Primitives
- Routing algorithms as tools for integrating social distancing with emergency evacuation
- Probabilistic design of optimal sequential decision-making algorithms in learning and control
- Parallel Exploration via Negatively Correlated Search
- Tracking-by-Trackers with a Distilled and Reinforced Model
- Reinforcement Learning for Systematic FX Trading
- On Wasserstein Reinforcement Learning and the Fokker-Planck equation
- ESceme: Vision-and-Language Navigation with Episodic Scene Memory
- Deep Reinforcement Learning With Macro-Actions
- Learning Adaptive Exploration Strategies in Dynamic Environments Through Informed Policy Regularization
- Control of nonlinear, complex and black-boxed greenhouse system with reinforcement learning
- Improving Model-Based Reinforcement Learning with Internal State Representations through Self-Supervision
- A review of motion planning algorithms for intelligent robotics
- Combining Subgoal Graphs with Reinforcement Learning to Build a Rational Pathfinder
- Entanglement engineering of optomechanical systems by reinforcement learning
- Entropy-Aware Model Initialization for Effective Exploration in Deep Reinforcement Learning
- Mutual influence between language and perception in multi-agent communication games
- Real-time visual tracking by deep reinforced decision making
- Comparing Deep Reinforcement Learning Algorithms in Two-Echelon Supply Chains
- Learning to Communicate Using Counterfactual Reasoning
- Investigating Recurrence and Eligibility Traces in Deep Q-Networks
- Attend, Adapt and Transfer: Attentive Deep Architecture for Adaptive Transfer from multiple sources in the same domain
- Flow-based Intrinsic Curiosity Module
- Boosting Deep Reinforcement Learning with Semantic Knowledge for Robotic Manipulators
- Analysing Factorizations of Action-Value Networks for Cooperative Multi-Agent Reinforcement Learning
- Compressed Federated Reinforcement Learning with a Generative Model
- Design of Restricted Normalizing Flow towards Arbitrary Stochastic Policy with Computational Efficiency
- Feasibility-Guided Learning for Robust Control in Constrained Optimal Control Problems
- Learning Invariances for Policy Generalization
- A short variational proof of equivalence between policy gradients and soft Q learning
- Globally Optimal Hierarchical Reinforcement Learning for Linearly-Solvable Markov Decision Processes
- Group-Agent Reinforcement Learning
- Optimal preference satisfaction for conflict-free joint decisions
- Fully Distributed Actor-Critic Architecture for Multitask Deep Reinforcement Learning
- DeepNav: Learning to Navigate Large Cities
- Capsule Network Performance with Autonomous Navigation
- MDP Playground: An Analysis and Debug Testbed for Reinforcement Learning
- Multi-agent Path Finding for Timed Tasks using Evolutionary Games
- Classification with Costly Features in Hierarchical Deep Sets
- Distributed Multi-Agent Deep Reinforcement Learning Framework for Whole-building HVAC Control
- A survey of benchmarking frameworks for reinforcement learning
- An advantage actor-critic algorithm for robotic motion planning in dense and dynamic scenarios
- Learning-based Hamilton-Jacobi-Bellman Methods for Optimal Control
- Tuning Synaptic Connections instead of Weights by Genetic Algorithm in Spiking Policy Network
- Sample Efficient Reinforcement Learning via Large Vision Language Model Distillation
- Using reinforcement learning to design an AI assistantfor a satisfying co-op experience
- Biological Blueprints for Next Generation AI Systems
- A Threshold-based Scheme for Reinforcement Learning in Neural Networks
- Conditioning of Reinforcement Learning Agents and its Policy Regularization Application
- FORLORN: A Framework for Comparing Offline Methods and Reinforcement Learning for Optimization of RAN Parameters
- Policy-Based Reinforcement Learning for Assortative Matching in Human Behavior Modeling
- Reinforcement Learning for Volt-Var Control: A Novel Two-stage Progressive Training Strategy
- Solving Atari Games Using Fractals And Entropy
- Deep RL Agent for a Real-Time Action Strategy Game