Benchmarking Deep Reinforcement Learning for Continuous Control

Abstract

Recently, researchers have made significant progress combining the advances in deep learning for learning feature representations with reinforcement learning. Some notable examples include training agents to play Atari games based on raw pixel data and to acquire advanced manipulation skills using raw sensory inputs. However, it has been difficult to quantify progress in the domain of continuous control due to the lack of a commonly adopted benchmark. In this work, we present a benchmark suite of continuous control tasks, including classic tasks like cart-pole swing-up, tasks with very high state and action dimensionality such as 3D humanoid locomotion, tasks with partial observations, and tasks with hierarchical structure. We report novel findings based on the systematic evaluation of a range of implemented reinforcement learning algorithms. Both the benchmark and reference implementations are released at https://github.com/rllab/rllab in order to facilitate experimental reproducibility and to encourage adoption by other researchers.

14 pages, ICML 2016

References in corpus (5)

• Continuous control with deep reinforcement learning
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• End-to-End Training of Deep Visuomotor Policies
• Learning Continuous Control Policies by Stochastic Value Gradients
• Memory-based control with recurrent neural networks

Cited by in corpus (227)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• A Brief Survey of Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Emergence of Locomotion Behaviours in Rich Environments
• Artificial Neural Networks trained through Deep Reinforcement Learning discover control strategies for active flow control
• Deep Reinforcement Learning: An Overview
• DeepMind Control Suite
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• VIME: Variational Information Maximizing Exploration
• Robust Adversarial Reinforcement Learning
• Parameter Space Noise for Exploration
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Model-Ensemble Trust-Region Policy Optimization
• Ray: A Distributed Framework for Emerging AI Applications
• Maximum a Posteriori Policy Optimisation
• Robust active flow control over a range of Reynolds numbers using an artificial neural network trained through deep reinforcement learning
• dm_control: Software and Tasks for Continuous Control
• Flow: A Modular Learning Framework for Mixed Autonomy Traffic
• Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
• Stochastic Neural Networks for Hierarchical Reinforcement Learning
• Meta-Reinforcement Learning of Structured Exploration Strategies
• Network-wide traffic signal control optimization using a multi-agent deep reinforcement learning
• Simple random search provides a competitive approach to reinforcement learning
• MOReL : Model-Based Offline Reinforcement Learning
• Temporal Difference Models: Model-Free Deep RL for Model-Based Control
• A Multi-Objective Deep Reinforcement Learning Framework
• EPOpt: Learning Robust Neural Network Policies Using Model Ensembles
• Emergent Complexity via Multi-Agent Competition
• Automatic Goal Generation for Reinforcement Learning Agents
• Modeling Human Driving Behavior through Generative Adversarial Imitation Learning
• Reverse Curriculum Generation for Reinforcement Learning
• Evolved Policy Gradients
• Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play
• Robust Deep Reinforcement Learning with Adversarial Attacks
• Levels of explainable artificial intelligence for human-aligned conversational explanations
• Deep Learning Approximation for Stochastic Control Problems
• Constrained Policy Optimization
• Assessing Generalization in Deep Reinforcement Learning
• Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
• Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning
• On the Sample Complexity of the Linear Quadratic Regulator
• Accelerated Methods for Deep Reinforcement Learning
• CEM-RL: Combining evolutionary and gradient-based methods for policy search
• A survey on intrinsic motivation in reinforcement learning
• Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Prediction
• Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines
• Artificial Intelligence and its Role in Near Future
• Towards Cognitive Exploration through Deep Reinforcement Learning for Mobile Robots
• Variational Option Discovery Algorithms
• GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms
• Active Learning in Robotics: A Review of Control Principles
• Stein Variational Policy Gradient
• A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation
• MO-MIX: Multi-Objective Multi-Agent Cooperative Decision-Making With Deep Reinforcement Learning
• Deep Reinforcement Learning Control of Quantum Cartpoles
• Lifelong Incremental Reinforcement Learning with Online Bayesian Inference
• Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learning
• Deep Reinforcement Learning for Event-Driven Multi-Agent Decision Processes
• An experimental evaluation of Deep Reinforcement Learning algorithms for HVAC control
• DPC-Net: Deep Pose Correction for Visual Localization
• One-Shot Reinforcement Learning for Robot Navigation with Interactive Replay
• Learning with AMIGo: Adversarially Motivated Intrinsic Goals
• Model-Free Control for Distributed Stream Data Processing using Deep Reinforcement Learning
• EvalAI: Towards Better Evaluation Systems for AI Agents
• Reinforcement learning of optimal active particle navigation
• The Mirage of Action-Dependent Baselines in Reinforcement Learning
• End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks
• OnSlicing: Online End-to-End Network Slicing with Reinforcement Learning
• Relative Entropy Regularized Policy Iteration
• Constrained Attractor Selection Using Deep Reinforcement Learning
• Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularization
• V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control
• learn2learn: A Library for Meta-Learning Research
• Hindsight policy gradients
• A Benchmarking Environment for Reinforcement Learning Based Task Oriented Dialogue Management
• Reinforcement Learning for Pivoting Task
• Investigating Generalisation in Continuous Deep Reinforcement Learning
• Recurrent Deterministic Policy Gradient Method for Bipedal Locomotion on Rough Terrain Challenge
• Decoupling Dynamics and Reward for Transfer Learning
• Truly Proximal Policy Optimization
• Mapping State Space using Landmarks for Universal Goal Reaching
• An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methods
• Natural Environment Benchmarks for Reinforcement Learning
• TensorFlow Agents: Efficient Batched Reinforcement Learning in TensorFlow
• On Learning Intrinsic Rewards for Policy Gradient Methods
• Continuous-Time Mean-Variance Portfolio Selection: A Reinforcement Learning Framework
• Optimal Control Via Neural Networks: A Convex Approach
• Trust-PCL: An Off-Policy Trust Region Method for Continuous Control
• A Survey of Deep Learning Techniques for Mobile Robot Applications
• Towards Generalization and Simplicity in Continuous Control
• An Adaptive Clipping Approach for Proximal Policy Optimization
• Noisy Natural Gradient as Variational Inference
• Learning Event-triggered Control from Data through Joint Optimization
• Multi-task Deep Reinforcement Learning with PopArt
• Pre-training with Non-expert Human Demonstration for Deep Reinforcement Learning
• Safe reinforcement learning for probabilistic reachability and safety specifications: A Lyapunov-based approach
• Clipped Action Policy Gradient
• Learning Functionally Decomposed Hierarchies for Continuous Control Tasks with Path Planning
• Evolutionary Quantum Architecture Search for Parametrized Quantum Circuits
• Scalable Multi-Agent Reinforcement Learning for Networked Systems with Average Reward
• Longitudinal Face Modeling via Temporal Deep Restricted Boltzmann Machines
• Agent-Agnostic Human-in-the-Loop Reinforcement Learning
• ORL: Reinforcement Learning Benchmarks for Online Stochastic Optimization Problems
• Meta-Learning through Hebbian Plasticity in Random Networks
• Mutual Alignment Transfer Learning
• The Faults in Our Pi Stars: Security Issues and Open Challenges in Deep Reinforcement Learning
• Boosting Trust Region Policy Optimization by Normalizing Flows Policy
• Value Functions Factorization with Latent State Information Sharing in Decentralized Multi-Agent Policy Gradients
• Learning a Low-dimensional Representation of a Safe Region for Safe Reinforcement Learning on Dynamical Systems
• Iroko: A Framework to Prototype Reinforcement Learning for Data Center Traffic Control
• Benchmark Environments for Multitask Learning in Continuous Domains
• GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning
• Delay-Aware Multi-Agent Reinforcement Learning for Cooperative and Competitive Environments
• Learning to Generate Pointing Gestures in Situated Embodied Conversational Agents
• Deep Reinforcement Learning Radio Control and Signal Detection with KeRLym, a Gym RL Agent
• Scalable Centralized Deep Multi-Agent Reinforcement Learning via Policy Gradients
• Hindsight Goal Ranking on Replay Buffer for Sparse Reward Environment
• Q-Learning for Continuous Actions with Cross-Entropy Guided Policies
• Real-Time Machine Learning: The Missing Pieces
• DropoutDAgger: A Bayesian Approach to Safe Imitation Learning
• Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learning
• Learning to Listen, Read, and Follow: Score Following as a Reinforcement Learning Game
• Time Limits in Reinforcement Learning
• Intervention Aided Reinforcement Learning for Safe and Practical Policy Optimization in Navigation
• Predictive-State Decoders: Encoding the Future into Recurrent Networks
• p-Meta: Towards On-device Deep Model Adaptation
• Procedural Content Generation: Better Benchmarks for Transfer Reinforcement Learning
• Combining Model-Based and Model-Free Methods for Nonlinear Control: A Provably Convergent Policy Gradient Approach
• OptLayer - Practical Constrained Optimization for Deep Reinforcement Learning in the Real World
• Skip Training for Multi-Agent Reinforcement Learning Controller for Industrial Wave Energy Converters
• SafeLife 1.0: Exploring Side Effects in Complex Environments
• Exploration by Distributional Reinforcement Learning
• Self-supervised Learning of Distance Functions for Goal-Conditioned Reinforcement Learning
• Comparing Deep Reinforcement Learning and Evolutionary Methods in Continuous Control
• Recurrent Predictive State Policy Networks
• Data-Efficient Policy Evaluation Through Behavior Policy Search
• Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning
• Coach: Exploiting Temporal Patterns for All-Resource Oversubscription in Cloud Platforms
• Deep Reinforcement Learning with a Combinatorial Action Space for Predicting Popular Reddit Threads
• Actor-critic versus direct policy search: a comparison based on sample complexity
• Automating Reinforcement Learning with Example-based Resets
• Evolving Inborn Knowledge For Fast Adaptation in Dynamic POMDP Problems
• Ready Policy One: World Building Through Active Learning
• Variational Deep Q Network
• Capability Iteration Network for Robot Path Planning
• A Deep Reinforcement Learning Approach for Traffic Signal Control Optimization
• Injecting Prior Knowledge for Transfer Learning into Reinforcement Learning Algorithms using Logic Tensor Networks
• The Eigenoption-Critic Framework
• A Hybrid Stochastic Policy Gradient Algorithm for Reinforcement Learning
• Safe Reinforcement Learning with Natural Language Constraints
• A Low Latency Adaptive Coding Spiking Framework for Deep Reinforcement Learning
• How to Make Deep RL Work in Practice
• Stochastic Variance Reduction for Policy Gradient Estimation
• Run, skeleton, run: skeletal model in a physics-based simulation
• Hierarchical Approaches for Reinforcement Learning in Parameterized Action Space
• Learning Extreme Hummingbird Maneuvers on Flapping Wing Robots
• From Complexity to Simplicity: Adaptive ES-Active Subspaces for Blackbox Optimization
• An Adaptable Approach to Learn Realistic Legged Locomotion without Examples
• HyperNCA: Growing Developmental Networks with Neural Cellular Automata
• Imitating Driver Behavior with Generative Adversarial Networks
• Catalyst.RL: A Distributed Framework for Reproducible RL Research
• Learning to Engage with Interactive Systems: A Field Study on Deep Reinforcement Learning in a Public Museum
• Analyzing the Variance of Policy Gradient Estimators for the Linear-Quadratic Regulator
• Bayesian Policy Optimization for Model Uncertainty
• Interactive Language Acquisition with One-shot Visual Concept Learning through a Conversational Game
• Realtime Scheduling and Power Allocation Using Deep Neural Networks
• Linear Representation Meta-Reinforcement Learning for Instant Adaptation
• Reinforcement Learning With Temporal Logic Rewards
• RLlib Flow: Distributed Reinforcement Learning is a Dataflow Problem
• Band-limited Soft Actor Critic Model
• Lyceum: An efficient and scalable ecosystem for robot learning
• GenSafe: A Generalizable Safety Enhancer for Safe Reinforcement Learning Algorithms Based on Reduced Order Markov Decision Process Model
• ANS: Adaptive Network Scaling for Deep Rectifier Reinforcement Learning Models
• DRL: Deep Reinforcement Learning for Intelligent Robot Control -- Concept, Literature, and Future
• Faster Algorithm and Sharper Analysis for Constrained Markov Decision Process
• Reset-Free Guided Policy Search: Efficient Deep Reinforcement Learning with Stochastic Initial States
• Enhanced Experience Replay Generation for Efficient Reinforcement Learning
• Role Playing Learning for Socially Concomitant Mobile Robot Navigation
• Learning Powerful Policies by Using Consistent Dynamics Model
• Learning Unmanned Aerial Vehicle Control for Autonomous Target Following
• Structured Monte Carlo Sampling for Nonisotropic Distributions via Determinantal Point Processes
• Reinforcement Learning for Intelligent Healthcare Systems: A Comprehensive Survey
• RLOC: Neurobiologically Inspired Hierarchical Reinforcement Learning Algorithm for Continuous Control of Nonlinear Dynamical Systems
• Optimization vs. Reinforcement Learning for Wirelessly Powered Sensor Networks
• Automatic Grouping of Redundant Sensors and Actuators Using Functional and Spatial Connections: Application to Muscle Grouping for Musculoskeletal Humanoids
• Learning Diverse Policies with Soft Self-Generated Guidance
• Switching Isotropic and Directional Exploration with Parameter Space Noise in Deep Reinforcement Learning
• Constrained Exploration and Recovery from Experience Shaping
• Automatic Representation for Lifetime Value Recommender Systems
• NavigationNet: A Large-scale Interactive Indoor Navigation Dataset
• Average-Reward Reinforcement Learning with Trust Region Methods
• Optimistic Distributionally Robust Policy Optimization
• Learning High-level Representations from Demonstrations
• Monte-Carlo Tree Search for Policy Optimization
• Expert Level control of Ramp Metering based on Multi-task Deep Reinforcement Learning
• Unlocking the Potential of Simulators: Design with RL in Mind
• oIRL: Robust Adversarial Inverse Reinforcement Learning with Temporally Extended Actions
• Can Reinforcement Learning for Continuous Control Generalize Across Physics Engines?
• Function Approximation for Reinforcement Learning Controller for Energy from Spread Waves
• Policy Optimization with Second-Order Advantage Information
• Fast Long-Term Multi-Scenario Prediction for Maneuver Planning at Unsignalized Intersections
• Tuning Synaptic Connections instead of Weights by Genetic Algorithm in Spiking Policy Network
• Learn Task First or Learn Human Partner First: A Hierarchical Task Decomposition Method for Human-Robot Cooperation
• Self-Imitation Learning for Robot Tasks with Sparse and Delayed Rewards
• Configuration Path Control
• A Hybrid Approach for Reinforcement Learning Using Virtual Policy Gradient for Balancing an Inverted Pendulum
• Autonomous Learning of Features for Control: Experiments with Embodied and Situated Agents
• VRGym: A Virtual Testbed for Physical and Interactive AI
• Learning Human Behaviors for Robot-Assisted Dressing
• A survey of benchmarking frameworks for reinforcement learning
• A Logarithmic Barrier Method For Proximal Policy Optimization
• Predictive Synthesis of Quantum Materials by Probabilistic Reinforcement Learning
• Policy Gradient Reinforcement Learning for Policy Represented by Fuzzy Rules: Application to Simulations of Speed Control of an Automobile
• Learning Robust and Adaptive Real-World Continuous Control Using Simulation and Transfer Learning
• The Role of Modularity and Neuro-Regulation for the Production of Multiple Behaviors
• Learning Time-Sensitive Strategies in Space Fortress
• Depth and nonlinearity induce implicit exploration for RL
• Region Growing Curriculum Generation for Reinforcement Learning
• Discrete linear-complexity reinforcement learning in continuous action spaces for Q-learning algorithms
• Mix and Mask Actor-Critic Methods
• End-to-End Race Driving with Deep Reinforcement Learning
• Reconciling Spatial and Temporal Abstractions for Goal Representation
• Wasserstein Flow Meets Replicator Dynamics: A Mean-Field Analysis of Representation Learning in Actor-Critic