Leveraging Procedural Generation to Benchmark Reinforcement Learning

Abstract

We introduce Procgen Benchmark, a suite of 16 procedurally generated game-like environments designed to benchmark both sample efficiency and generalization in reinforcement learning. We believe that the community will benefit from increased access to high quality training environments, and we provide detailed experimental protocols for using this benchmark. We empirically demonstrate that diverse environment distributions are essential to adequately train and evaluate RL agents, thereby motivating the extensive use of procedural content generation. We then use this benchmark to investigate the effects of scaling model size, finding that larger models significantly improve both sample efficiency and generalization.

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation
• Assessing Generalization in Deep Reinforcement Learning
• Generalization and Regularization in DQN
• A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning
• Gotta Learn Fast: A New Benchmark for Generalization in RL
• Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learning
• Behaviour Suite for Reinforcement Learning
• Natural Environment Benchmarks for Reinforcement Learning

Cited by in corpus (53)

• Unity: A General Platform for Intelligent Agents
• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Learning explanations that are hard to vary
• Decoupling Representation Learning from Reinforcement Learning
• Automatic Data Augmentation for Generalization in Deep Reinforcement Learning
• Phasic Policy Gradient
• Improving Generalization in Reinforcement Learning with Mixture Regularization
• The NetHack Learning Environment
• Learning with AMIGo: Adversarially Motivated Intrinsic Goals
• ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations
• GraphWorld: Fake Graphs Bring Real Insights for GNNs
• Sample-efficient Reinforcement Learning Representation Learning with Curiosity Contrastive Forward Dynamics Model
• A Neural Scaling Law from the Dimension of the Data Manifold
• Generalization in Reinforcement Learning by Soft Data Augmentation
• Obtaining Robust Control and Navigation Policies for Multi-Robot Navigation via Deep Reinforcement Learning
• Using a thousand optimization tasks to learn hyperparameter search strategies
• SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies
• Invariant Policy Optimization: Towards Stronger Generalization in Reinforcement Learning
• Extending Environments To Measure Self-Reflection In Reinforcement Learning
• Reinforcement Learning with Dual-Observation for General Video Game Playing
• Procedural Content Generation: Better Benchmarks for Transfer Reinforcement Learning
• Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability
• Transient Non-Stationarity and Generalisation in Deep Reinforcement Learning
• Rotation, Translation, and Cropping for Zero-Shot Generalization
• Measuring Visual Generalization in Continuous Control from Pixels
• Fever Basketball: A Complex, Flexible, and Asynchronized Sports Game Environment for Multi-agent Reinforcement Learning
• Reinforcement Learning Generalization with Surprise Minimization
• An Empirical Investigation of Representation Learning for Imitation
• rl_reach: Reproducible Reinforcement Learning Experiments for Robotic Reaching Tasks
• The MAGICAL Benchmark for Robust Imitation
• Domain Adversarial Reinforcement Learning
• Refactoring Policy for Compositional Generalizability using Self-Supervised Object Proposals
• Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environments
• Reinforcement Learning with Success Induced Task Prioritization
• CROP: Towards Distributional-Shift Robust Reinforcement Learning using Compact Reshaped Observation Processing
• Decoupling Value and Policy for Generalization in Reinforcement Learning
• CARL: A Benchmark for Contextual and Adaptive Reinforcement Learning
• A Brief Look at Generalization in Visual Meta-Reinforcement Learning
• Evolutionary Selective Imitation: Interpretable Agents by Imitation Learning Without a Demonstrator
• Sparse Attention Guided Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement Learning
• Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement Learning
• Discovering Generalizable Skills via Automated Generation of Diverse Tasks
• Instance based Generalization in Reinforcement Learning
• Meta Arcade: A Configurable Environment Suite for Meta-Learning
• AMaze: An intuitive benchmark generator for fast prototyping of generalizable agents
• Deep Policy Networks for NPC Behaviors that Adapt to Changing Design Parameters in Roguelike Games
• MDP Playground: An Analysis and Debug Testbed for Reinforcement Learning
• OpEvo: An Evolutionary Method for Tensor Operator Optimization
• Diagnosing and exploiting the computational demands of videos games for deep reinforcement learning
• Enhanced Scene Specificity with Sparse Dynamic Value Estimation
• Batch size-invariance for policy optimization