Deep Reinforcement Learning at the Edge of the Statistical Precipice

Abstract

Deep reinforcement learning (RL) algorithms are predominantly evaluated by comparing their relative performance on a large suite of tasks. Most published results on deep RL benchmarks compare point estimates of aggregate performance such as mean and median scores across tasks, ignoring the statistical uncertainty implied by the use of a finite number of training runs. Beginning with the Arcade Learning Environment (ALE), the shift towards computationally-demanding benchmarks has led to the practice of evaluating only a small number of runs per task, exacerbating the statistical uncertainty in point estimates. In this paper, we argue that reliable evaluation in the few run deep RL regime cannot ignore the uncertainty in results without running the risk of slowing down progress in the field. We illustrate this point using a case study on the Atari 100k benchmark, where we find substantial discrepancies between conclusions drawn from point estimates alone versus a more thorough statistical analysis. With the aim of increasing the field's confidence in reported results with a handful of runs, we advocate for reporting interval estimates of aggregate performance and propose performance profiles to account for the variability in results, as well as present more robust and efficient aggregate metrics, such as interquartile mean scores, to achieve small uncertainty in results. Using such statistical tools, we scrutinize performance evaluations of existing algorithms on other widely used RL benchmarks including the ALE, Procgen, and the DeepMind Control Suite, again revealing discrepancies in prior comparisons. Our findings call for a change in how we evaluate performance in deep RL, for which we present a more rigorous evaluation methodology, accompanied with an open-source library rliable, to prevent unreliable results from stagnating the field.

Outstanding Paper Award at NeurIPS 2021. Website: https://agarwl.github.io/rliable. 28 Pages, 33 Figures

References in corpus (22)

• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Agent57: Outperforming the Atari Human Benchmark
• Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
• Dopamine: A Research Framework for Deep Reinforcement Learning
• Distributional Reinforcement Learning with Quantile Regression
• Automatic Data Augmentation for Generalization in Deep Reinforcement Learning
• Phasic Policy Gradient
• Improving Generalization in Reinforcement Learning with Mixture Regularization
• Measuring the Reliability of Reinforcement Learning Algorithms
• Munchausen Reinforcement Learning
• Evaluating the Performance of Reinforcement Learning Algorithms
• Return-Based Contrastive Representation Learning for Reinforcement Learning
• The MultiBERTs: BERT Reproductions for Robustness Analysis
• Let's Play Again: Variability of Deep Reinforcement Learning Agents in Atari Environments
• D2RL: Deep Dense Architectures in Reinforcement Learning
• APS: Active Pretraining with Successor Features
• Synthetic Returns for Long-Term Credit Assignment
• Behavior From the Void: Unsupervised Active Pre-Training
• The Benchmark Lottery
• Return-based Scaling: Yet Another Normalisation Trick for Deep RL
• Significant Improvements over the State of the Art? A Case Study of the MS MARCO Document Ranking Leaderboard
• Catalyst.RL: A Distributed Framework for Reproducible RL Research

Cited by in corpus (4)

• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Beyond Ads: Sequential Decision-Making Algorithms in Law and Public Policy
• A Domain-Agnostic Approach for Characterization of Lifelong Learning Systems
• A Survey of Learning Criteria Going Beyond the Usual Risk