Measuring Progress in Deep Reinforcement Learning Sample Efficiency

Abstract

Sampled environment transitions are a critical input to deep reinforcement learning (DRL) algorithms. Current DRL benchmarks often allow for the cheap and easy generation of large amounts of samples such that perceived progress in DRL does not necessarily correspond to improved sample efficiency. As simulating real world processes is often prohibitively hard and collecting real world experience is costly, sample efficiency is an important indicator for economically relevant applications of DRL. We investigate progress in sample efficiency on Atari games and continuous control tasks by comparing the number of samples that a variety of algorithms need to reach a given performance level according to training curves in the corresponding publications. We find exponential progress in sample efficiency with estimated doubling times of around 10 to 18 months on Atari, 5 to 24 months on state-based continuous control and of around 4 to 9 months on pixel-based continuous control depending on the specific task and performance level.

26 pages, 6 figures, 5 tables

References in corpus (57)

• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Addressing Function Approximation Error in Actor-Critic Methods
• Prioritized Experience Replay
• Dueling Network Architectures for Deep Reinforcement Learning
• Scaling Laws for Neural Language Models
• Dota 2 with Large Scale Deep Reinforcement Learning
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Noisy Networks for Exploration
• DeepMind Control Suite
• Distributed Prioritized Experience Replay
• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Model-Based Reinforcement Learning for Atari
• Deep Reinforcement Learning that Matters
• Learning Latent Dynamics for Planning from Pixels
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Distributed Distributional Deterministic Policy Gradients
• Challenges of Real-World Reinforcement Learning
• Implicit Quantile Networks for Distributional Reinforcement Learning
• Reinforcement Learning with Augmented Data
• A Distributional Perspective on Reinforcement Learning
• Agent57: Outperforming the Atari Human Benchmark
• Maximum a Posteriori Policy Optimisation
• Sample Efficient Actor-Critic with Experience Replay
• Go-Explore: a New Approach for Hard-Exploration Problems
• Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
• Distributional Reinforcement Learning with Quantile Regression
• Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion
• End-to-End Deep Reinforcement Learning for Lane Keeping Assist
• Dream to Control: Learning Behaviors by Latent Imagination
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• Model-Based Reinforcement Learning via Meta-Policy Optimization
• When to use parametric models in reinforcement learning?
• Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic
• Combining policy gradient and Q-learning
• Measuring the Algorithmic Efficiency of Neural Networks
• Exploring Model-based Planning with Policy Networks
• Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model
• Planning to Explore via Self-Supervised World Models
• Data-Efficient Reinforcement Learning with Self-Predictive Representations
• SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning
• InfoBot: Transfer and Exploration via the Information Bottleneck
• Relative Entropy Regularized Policy Iteration
• Meta-Gradient Reinforcement Learning with an Objective Discovered Online
• State of the Art Control of Atari Games Using Shallow Reinforcement Learning
• SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference
• Trust-PCL: An Off-Policy Trust Region Method for Continuous Control
• Clipped Action Policy Gradient
• REPLAB: A Reproducible Low-Cost Arm Benchmark Platform for Robotic Learning
• Is Deep Reinforcement Learning Really Superhuman on Atari? Leveling the playing field
• Convergence Rates of Accelerated Markov Gradient Descent with Applications in Reinforcement Learning
• Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstruction
• Soft Policy Gradient Method for Maximum Entropy Deep Reinforcement Learning