Deep Reinforcement Learning that Matters

Abstract

In recent years, significant progress has been made in solving challenging problems across various domains using deep reinforcement learning (RL). Reproducing existing work and accurately judging the improvements offered by novel methods is vital to sustaining this progress. Unfortunately, reproducing results for state-of-the-art deep RL methods is seldom straightforward. In particular, non-determinism in standard benchmark environments, combined with variance intrinsic to the methods, can make reported results tough to interpret. Without significance metrics and tighter standardization of experimental reporting, it is difficult to determine whether improvements over the prior state-of-the-art are meaningful. In this paper, we investigate challenges posed by reproducibility, proper experimental techniques, and reporting procedures. We illustrate the variability in reported metrics and results when comparing against common baselines and suggest guidelines to make future results in deep RL more reproducible. We aim to spur discussion about how to ensure continued progress in the field by minimizing wasted effort stemming from results that are non-reproducible and easily misinterpreted.

Accepted to the Thirthy-Second AAAI Conference On Artificial Intelligence (AAAI), 2018

Cited by in corpus (203)

• Deep learning for molecular design - a review of the state of the art
• explAIner: A Visual Analytics Framework for Interactive and Explainable Machine Learning
• Challenges of Real-World Reinforcement Learning
• Reinforcement Learning with Augmented Data
• Benchmarking Model-Based Reinforcement Learning
• Comparison of Deep Reinforcement Learning and Model Predictive Control for Adaptive Cruise Control
• CURL: Contrastive Unsupervised Representations for Reinforcement Learning
• Realistic Evaluation of Deep Semi-Supervised Learning Algorithms
• Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning
• A Minimalist Approach to Offline Reinforcement Learning
• robosuite: A Modular Simulation Framework and Benchmark for Robot Learning
• A Survey on Device Behavior Fingerprinting: Data Sources, Techniques, Application Scenarios, and Datasets
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• The Fundamental Principles of Reproducibility
• Learn to Schedule (LEASCH): A Deep reinforcement learning approach for radio resource scheduling in the 5G MAC layer
• Best Practices for Scientific Research on Neural Architecture Search
• On the Value of Out-of-Distribution Testing: An Example of Goodhart's Law
• What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
• ChainerRL: A Deep Reinforcement Learning Library
• Iterative Reinforcement Learning Based Design of Dynamic Locomotion Skills for Cassie
• Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning
• SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards
• An empirical investigation of the challenges of real-world reinforcement learning
• Data-Efficient Reinforcement Learning with Self-Predictive Representations
• MinAtar: An Atari-Inspired Testbed for Thorough and Reproducible Reinforcement Learning Experiments
• Conditional Equivalence Testing: an alternative remedy for publication bias
• Learning to Locomote: Understanding How Environment Design Matters for Deep Reinforcement Learning
• Deep Reinforcement Learning at the Edge of the Statistical Precipice
• Boosting Soft Actor-Critic: Emphasizing Recent Experience without Forgetting the Past
• Accounting for Variance in Machine Learning Benchmarks
• Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations
• Deep reinforcement learning-designed radiofrequency waveform in MRI
• Hyp-RL : Hyperparameter Optimization by Reinforcement Learning
• AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Rates
• Investigating Generalisation in Continuous Deep Reinforcement Learning
• DeepRacer: Educational Autonomous Racing Platform for Experimentation with Sim2Real Reinforcement Learning
• Cryptocurrency Trading: A Comprehensive Survey
• Deep Reinforcement Learning for Closed-Loop Blood Glucose Control
• The Scientific Method in the Science of Machine Learning
• Randomized Ensembled Double Q-Learning: Learning Fast Without a Model
• Learning to Reach Goals via Iterated Supervised Learning
• Revisiting State Augmentation methods for Reinforcement Learning with Stochastic Delays
• Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learning
• TriFinger: An Open-Source Robot for Learning Dexterity
• Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings
• Benchmarking Bonus-Based Exploration Methods on the Arcade Learning Environment
• On Bonus-Based Exploration Methods in the Arcade Learning Environment
• REPLAB: A Reproducible Low-Cost Arm Benchmark Platform for Robotic Learning
• Optimistic Reinforcement Learning by Forward Kullback-Leibler Divergence Optimization
• Hierarchical Reinforcement Learning via Advantage-Weighted Information Maximization
• Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Research
• Show Your Work: Improved Reporting of Experimental Results
• Is Deep Reinforcement Learning Really Superhuman on Atari? Leveling the playing field
• Multiplex Communities and the Emergence of International Conflict
• What Matters for Adversarial Imitation Learning?
• Training Agents using Upside-Down Reinforcement Learning
• Adaptive Power System Emergency Control using Deep Reinforcement Learning
• Regularization Matters in Policy Optimization
• On the Reproducibility of Neural Network Predictions
• Dynamics Generalization via Information Bottleneck in Deep Reinforcement Learning
• Using AntiPatterns to avoid MLOps Mistakes
• On the model-based stochastic value gradient for continuous reinforcement learning
• Modern Deep Reinforcement Learning Algorithms
• Q-Learning for Continuous Actions with Cross-Entropy Guided Policies
• MBRL-Lib: A Modular Library for Model-based Reinforcement Learning
• Playing Atari with Hybrid Quantum-Classical Reinforcement Learning
• Hierarchical Reinforcement Learning Framework for Stochastic Spaceflight Campaign Design
• Can Increasing Input Dimensionality Improve Deep Reinforcement Learning?
• Flight Controller Synthesis Via Deep Reinforcement Learning
• How You Act Tells a Lot: Privacy-Leakage Attack on Deep Reinforcement Learning
• Stochastic Recursive Momentum for Policy Gradient Methods
• Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks
• Optimizer Benchmarking Needs to Account for Hyperparameter Tuning
• Observation Space Matters: Benchmark and Optimization Algorithm
• An initial attempt of combining visual selective attention with deep reinforcement learning
• Neural Temporal-Difference and Q-Learning Provably Converge to Global Optima
• Striving for Simplicity and Performance in Off-Policy DRL: Output Normalization and Non-Uniform Sampling
• Revisiting Design Choices in Proximal Policy Optimization
• Simplifying Deep Reinforcement Learning via Self-Supervision
• PIC: Permutation Invariant Critic for Multi-Agent Deep Reinforcement Learning
• Hyperparameter Auto-tuning in Self-Supervised Robotic Learning
• Reward Estimation for Variance Reduction in Deep Reinforcement Learning
• Fast Efficient Hyperparameter Tuning for Policy Gradients
• Reinforced Epidemic Control: Saving Both Lives and Economy
• The effect of Target Normalization and Momentum on Dying ReLU
• Training Larger Networks for Deep Reinforcement Learning
• An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replay
• XIRL: Cross-embodiment Inverse Reinforcement Learning
• Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergences
• Model Selection's Disparate Impact in Real-World Deep Learning Applications
• The Effect of Multi-step Methods on Overestimation in Deep Reinforcement Learning
• Improving Performance in Reinforcement Learning by Breaking Generalization in Neural Networks
• The Benchmark Lottery
• Neural-encoding Human Experts' Domain Knowledge to Warm Start Reinforcement Learning
• Imitation-Projected Programmatic Reinforcement Learning
• Disentangling Adaptive Gradient Methods from Learning Rates
• SmartChoices: Hybridizing Programming and Machine Learning
• Population-coding and Dynamic-neurons improved Spiking Actor Network for Reinforcement Learning
• rl_reach: Reproducible Reinforcement Learning Experiments for Robotic Reaching Tasks
• Exploratory State Representation Learning
• MEPG: A Minimalist Ensemble Policy Gradient Framework for Deep Reinforcement Learning
• O2A: One-shot Observational learning with Action vectors
• When Is Generalizable Reinforcement Learning Tractable?
• Nondeterminism and Instability in Neural Network Optimization
• Learning Vision-based Reactive Policies for Obstacle Avoidance
• Policy Optimization for Markovian Jump Linear Quadratic Control: Gradient-Based Methods and Global Convergence
• Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Models
• Catalyst.RL: A Distributed Framework for Reproducible RL Research
• How Much Do Unstated Problem Constraints Limit Deep Robotic Reinforcement Learning?
• Verification in the Loop: Correct-by-Construction Control Learning with Reach-avoid Guarantees
• Learning Policies from Self-Play with Policy Gradients and MCTS Value Estimates
• Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environments
• Hyperparameter Optimization Is Deceiving Us, and How to Stop It
• Offline Reinforcement Learning with Pseudometric Learning
• Learning to Engage with Interactive Systems: A Field Study on Deep Reinforcement Learning in a Public Museum
• Policy Learning of MDPs with Mixed Continuous/Discrete Variables: A Case Study on Model-Free Control of Markovian Jump Systems
• Replication Markets: Results, Lessons, Challenges and Opportunities in AI Replication
• Playing Minecraft with Behavioural Cloning
• Intelligent Coordination among Multiple Traffic Intersections Using Multi-Agent Reinforcement Learning
• A Pragmatic Look at Deep Imitation Learning
• Sequoia: A Software Framework to Unify Continual Learning Research
• f-IRL: Inverse Reinforcement Learning via State Marginal Matching
• Evaluating the progress of Deep Reinforcement Learning in the real world: aligning domain-agnostic and domain-specific research
• Leveraging Human Guidance for Deep Reinforcement Learning Tasks
• Run, Forest, Run? On Randomization and Reproducibility in Predictive Software Engineering
• Qgraph-bounded Q-learning: Stabilizing Model-Free Off-Policy Deep Reinforcement Learning
• Quantity vs. Quality: On Hyperparameter Optimization for Deep Reinforcement Learning
• Challenges for cognitive decoding using deep learning methods
• P3O: Policy-on Policy-off Policy Optimization
• On the Search for Feedback in Reinforcement Learning
• Stacked Auto Encoder Based Deep Reinforcement Learning for Online Resource Scheduling in Large-Scale MEC Networks
• Scenic4RL: Programmatic Modeling and Generation of Reinforcement Learning Environments
• A Survey on Reproducibility by Evaluating Deep Reinforcement Learning Algorithms on Real-World Robots
• IEO: Intelligent Evolutionary Optimisation for Hyperparameter Tuning
• Identifying Critical States by the Action-Based Variance of Expected Return
• Multi-Path Policy Optimization
• Bayesian Optimization for Iterative Learning
• MobileVisFixer: Tailoring Web Visualizations for Mobile Phones Leveraging an Explainable Reinforcement Learning Framework
• Co-training for Policy Learning
• Efficient Reinforcement Learning for StarCraft by Abstract Forward Models and Transfer Learning
• Aggressive Q-Learning with Ensembles: Achieving Both High Sample Efficiency and High Asymptotic Performance
• Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRL
• Recruitment-imitation Mechanism for Evolutionary Reinforcement Learning
• Asynchronous Episodic Deep Deterministic Policy Gradient: Towards Continuous Control in Computationally Complex Environments
• Avoiding Tampering Incentives in Deep RL via Decoupled Approval
• An Intelligent Energy Management Framework for Hybrid-Electric Propulsion Systems Using Deep Reinforcement Learning
• The MIS Check-Dam Dataset for Object Detection and Instance Segmentation Tasks
• CLEVA-Compass: A Continual Learning EValuation Assessment Compass to Promote Research Transparency and Comparability
• A Distance-based Anomaly Detection Framework for Deep Reinforcement Learning
• Sample Efficient Ensemble Learning with Catalyst.RL
• MLE-guided parameter search for task loss minimization in neural sequence modeling
• Dual Reconstruction: a Unifying Objective for Semi-Supervised Neural Machine Translation
• A Differential Game Theoretic Neural Optimizer for Training Residual Networks
• Technology Readiness Levels for AI & ML
• Novel Policy Seeking with Constrained Optimization
• Model-Free Episodic Control with State Aggregation
• AI Driven Heterogeneous MEC System with UAV Assistance for Dynamic Environment -- Challenges and Solutions
• Approximating two value functions instead of one: towards characterizing a new family of Deep Reinforcement Learning algorithms
• Individual predictions matter: Assessing the effect of data ordering in training fine-tuned CNNs for medical imaging
• Efficient Robotic Task Generalization Using Deep Model Fusion Reinforcement Learning
• Discovering Differential Features: Adversarial Learning for Information Credibility Evaluation
• Learning to Learn and Predict: A Meta-Learning Approach for Multi-Label Classification
• D2C 2.0: Decoupled Data-Based Approach for Learning to Control Stochastic Nonlinear Systems via Model-Free ILQR
• Reparameterized Variational Divergence Minimization for Stable Imitation
• Improving the Exploration of Deep Reinforcement Learning in Continuous Domains using Planning for Policy Search
• The Text-Based Adventure AI Competition
• A survey of benchmarking frameworks for reinforcement learning
• Learning Policies through Quantile Regression
• General Characterization of Agents by States they Visit
• A Learning-Based Computational Impact Time Guidance
• Pessimistic Model Selection for Offline Deep Reinforcement Learning
• Bellman: A Toolbox for Model-Based Reinforcement Learning in TensorFlow
• DECoVaC: Design of Experiments with Controlled Variability Components
• Differentiable Sampling with Flexible Reference Word Order for Neural Machine Translation
• Robust Risk-Sensitive Reinforcement Learning Agents for Trading Markets
• HyperSched: Dynamic Resource Reallocation for Model Development on a Deadline
• PA-Cache: Evolving Learning-Based Popularity-Aware Content Caching in Edge Networks
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• Large scale continuous-time mean-variance portfolio allocation via reinforcement learning
• DERAIL: Diagnostic Environments for Reward And Imitation Learning
• Reinforcement learning with world model
• Integrated Benchmarking and Design for Reproducible and Accessible Evaluation of Robotic Agents
• Offline RL With Resource Constrained Online Deployment
• Improved Memories Learning
• Understanding the Effect of Stochasticity in Policy Optimization
• A Novel Update Mechanism for Q-Networks Based On Extreme Learning Machines
• Weighted Entropy Modification for Soft Actor-Critic
• Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning
• Going Beyond Linear RL: Sample Efficient Neural Function Approximation
• Graphs for deep learning representations
• Relational Mimic for Visual Adversarial Imitation Learning
• Optimizing Quantum Variational Circuits with Deep Reinforcement Learning
• Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters
• Model-Free $\mu$ Synthesis via Adversarial Reinforcement Learning
• FiDi-RL: Incorporating Deep Reinforcement Learning with Finite-Difference Policy Search for Efficient Learning of Continuous Control
• On The Transferability of Deep-Q Networks
• Count-Based Temperature Scheduling for Maximum Entropy Reinforcement Learning
• A Dynamics Perspective of Pursuit-Evasion Games of Intelligent Agents with the Ability to Learn
• DCUR: Data Curriculum for Teaching via Samples with Reinforcement Learning
• An initial investigation on optimizing tandem speaker verification and countermeasure systems using reinforcement learning
• RLCache: Automated Cache Management Using Reinforcement Learning
• GrowSpace: Learning How to Shape Plants
• Weighting NTBEA for Game AI Optimisation