Measuring Progress in Deep Reinforcement Learning Sample Efficiency

Abstract

Sampled environment transitions are a critical input to deep reinforcement learning (DRL) algorithms. Current DRL benchmarks often allow for the cheap and easy generation of large amounts of samples such that perceived progress in DRL does not necessarily correspond to improved sample efficiency. As simulating real world processes is often prohibitively hard and collecting real world experience is costly, sample efficiency is an important indicator for economically relevant applications of DRL. We investigate progress in sample efficiency on Atari games and continuous control tasks by comparing the number of samples that a variety of algorithms need to reach a given performance level according to training curves in the corresponding publications. We find exponential progress in sample efficiency with estimated doubling times of around 10 to 18 months on Atari, 5 to 24 months on state-based continuous control and of around 4 to 9 months on pixel-based continuous control depending on the specific task and performance level.

26 pages, 6 figures, 5 tables

References in corpus (16)

• Scaling Laws for Neural Language Models
• DeepMind Control Suite
• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Challenges of Real-World Reinforcement Learning
• A Distributional Perspective on Reinforcement Learning
• Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
• Distributional Reinforcement Learning with Quantile Regression
• End-to-End Deep Reinforcement Learning for Lane Keeping Assist
• Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic
• Measuring the Algorithmic Efficiency of Neural Networks
• Exploring Model-based Planning with Policy Networks
• Planning to Explore via Self-Supervised World Models
• Meta-Gradient Reinforcement Learning with an Objective Discovered Online
• SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference
• REPLAB: A Reproducible Low-Cost Arm Benchmark Platform for Robotic Learning