Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control

Abstract

Policy gradient methods in reinforcement learning have become increasingly prevalent for state-of-the-art performance in continuous control tasks. Novel methods typically benchmark against a few key algorithms such as deep deterministic policy gradients and trust region policy optimization. As such, it is important to present and use consistent baselines experiments. However, this can be difficult due to general variance in the algorithms, hyper-parameter tuning, and environment stochasticity. We investigate and discuss: the significance of hyper-parameters in policy gradients for continuous control, general variance in the algorithms, and reproducibility of reported results. We provide guidelines on reporting novel results as comparisons against baseline methods such that future researchers can make informed decisions when investigating novel methods.

Accepted to Reproducibility in Machine Learning Workshop, ICML'17

References in corpus (9)

• Continuous control with deep reinforcement learning
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• End-to-End Training of Deep Visuomotor Policies
• Benchmarking Deep Reinforcement Learning for Continuous Control
• VIME: Variational Information Maximizing Exploration
• EPOpt: Learning Robust Neural Network Policies Using Model Ensembles
• Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic
• Discrete Sequential Prediction of Continuous Actions for Deep RL
• Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning

Cited by in corpus (72)

• Deep Reinforcement Learning: An Overview
• DeepMind Control Suite
• Benchmarking Model-Based Reinforcement Learning
• Simple random search provides a competitive approach to reinforcement learning
• What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms
• ChainerRL: A Deep Reinforcement Learning Library
• Discrete Sequential Prediction of Continuous Actions for Deep RL
• How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments
• An experimental evaluation of Deep Reinforcement Learning algorithms for HVAC control
• Deep Reinforcement Learning at the Edge of the Statistical Precipice
• Measuring the Reliability of Reinforcement Learning Algorithms
• Boosting Soft Actor-Critic: Emphasizing Recent Experience without Forgetting the Past
• On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning
• Deterministic Implementations for Reproducibility in Deep Reinforcement Learning
• A Reinforcement Learning Approach for Transient Control of Liquid Rocket Engines
• Deep Reinforcement Learning for Closed-Loop Blood Glucose Control
• On the Global Convergence of Actor-Critic: A Case for Linear Quadratic Regulator with Ergodic Cost
• Randomized Ensembled Double Q-Learning: Learning Fast Without a Model
• One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control
• A Closer Look at Deep Policy Gradients
• Accuracy-based Curriculum Learning in Deep Reinforcement Learning
• CACTO: Continuous Actor-Critic with Trajectory Optimization -- Towards global optimality
• A Tour of Reinforcement Learning: The View from Continuous Control
• What Matters for Adversarial Imitation Learning?
• On the model-based stochastic value gradient for continuous reinforcement learning
• Where Did My Optimum Go?: An Empirical Analysis of Gradient Descent Optimization in Policy Gradient Methods
• Q-Learning for Continuous Actions with Cross-Entropy Guided Policies
• Remember and Forget for Experience Replay
• Observation Space Matters: Benchmark and Optimization Algorithm
• DropoutDAgger: A Bayesian Approach to Safe Imitation Learning
• Striving for Simplicity and Performance in Off-Policy DRL: Output Normalization and Non-Uniform Sampling
• Simplifying Deep Reinforcement Learning via Self-Supervision
• Adversarial Imitation Learning via Random Search
• Map-based Multi-Policy Reinforcement Learning: Enhancing Adaptability of Robots by Deep Reinforcement Learning
• ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
• Deep Model-Based Reinforcement Learning for High-Dimensional Problems, a Survey
• How to Make Deep RL Work in Practice
• Nondeterminism and Instability in Neural Network Optimization
• SLM Lab: A Comprehensive Benchmark and Modular Software Framework for Reproducible Deep Reinforcement Learning
• Catalyst.RL: A Distributed Framework for Reproducible RL Research
• Automatic Neural Network Compression by Sparsity-Quantization Joint Learning: A Constrained Optimization-based Approach
• Quantity vs. Quality: On Hyperparameter Optimization for Deep Reinforcement Learning
• A Study of Checkpointing in Large Scale Training of Deep Neural Networks
• Sample-Efficient Automated Deep Reinforcement Learning
• A Survey on Reproducibility by Evaluating Deep Reinforcement Learning Algorithms on Real-World Robots
• Aggressive Q-Learning with Ensembles: Achieving Both High Sample Efficiency and High Asymptotic Performance
• Augmented Random Search for Quadcopter Control: An alternative to Reinforcement Learning
• Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRL
• Kernel-based diffusion approximated Markov decision processes for autonomous navigation and control on unstructured terrains
• On the Search for Feedback in Reinforcement Learning
• Recruitment-imitation Mechanism for Evolutionary Reinforcement Learning
• D2C 2.0: Decoupled Data-Based Approach for Learning to Control Stochastic Nonlinear Systems via Model-Free ILQR
• Sample Efficient Ensemble Learning with Catalyst.RL
• On Proximal Policy Optimization's Heavy-tailed Gradients
• Reinforcement Learning for Multi-Objective Optimization of Online Decisions in High-Dimensional Systems
• Randomized Adversarial Imitation Learning for Autonomous Driving
• Learning Policies through Quantile Regression
• Comparing Reinforcement Learning and Human Learning using the Game of Hidden Rules
• A survey of benchmarking frameworks for reinforcement learning
• DERAIL: Diagnostic Environments for Reward And Imitation Learning
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• Analyzing Reinforcement Learning Benchmarks with Random Weight Guessing
• Regularly Updated Deterministic Policy Gradient Algorithm
• Steps Towards Best Practices For Robot Videos
• Kernel Taylor-Based Value Function Approximation for Continuous-State Markov Decision Processes
• Learning Physical-Layer Communication with Quantized Feedback
• Doubly Robust Off-Policy Actor-Critic Algorithms for Reinforcement Learning
• A Methodology for the Development of RL-Based Adaptive Traffic Signal Controllers
• Randomized Policy Learning for Continuous State and Action MDPs
• Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning