Quantifying Generalization in Reinforcement Learning

Abstract

In this paper, we investigate the problem of overfitting in deep reinforcement learning. Among the most common benchmarks in RL, it is customary to use the same environments for both training and testing. This practice offers relatively little insight into an agent's ability to generalize. We address this issue by using procedurally generated environments to construct distinct training and test sets. Most notably, we introduce a new environment called CoinRun, designed as a benchmark for generalization in RL. Using CoinRun, we find that agents overfit to surprisingly large training sets. We then show that deeper convolutional architectures improve generalization, as do methods traditionally found in supervised learning, including L2 regularization, dropout, data augmentation and batch normalization.

References in corpus (12)

• Playing Atari with Deep Reinforcement Learning
• Improved Regularization of Convolutional Neural Networks with Cutout
• AutoAugment: Learning Augmentation Policies from Data
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• A Study on Overfitting in Deep Reinforcement Learning
• Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World
• Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation
• Assessing Generalization in Deep Reinforcement Learning
• Generalization and Regularization in DQN
• A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning
• Gotta Learn Fast: A New Benchmark for Generalization in RL
• Towards Understanding Regularization in Batch Normalization

Cited by in corpus (79)

• Dota 2 with Large Scale Deep Reinforcement Learning
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• MOPO: Model-based Offline Policy Optimization
• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• Resource Management in Wireless Networks via Multi-Agent Deep Reinforcement Learning
• A Survey of Deep Reinforcement Learning in Video Games
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoning
• Assessing Generalization in Deep Reinforcement Learning
• Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP
• Generalization and Regularization in DQN
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Unsolved Problems in ML Safety
• Learning to Generalize from Sparse and Underspecified Rewards
• Imitating Latent Policies from Observation
• Automatic Data Augmentation for Generalization in Deep Reinforcement Learning
• Improving Generalization in Reinforcement Learning with Mixture Regularization
• TAP-Net: Transport-and-Pack using Reinforcement Learning
• Learning the Travelling Salesperson Problem Requires Rethinking Generalization
• Invariant Causal Prediction for Block MDPs
• Investigating Generalisation in Continuous Deep Reinforcement Learning
• Active Domain Randomization
• CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning
• Off-Policy Evaluation via Off-Policy Classification
• CybORG: An Autonomous Cyber Operations Research Gym
• Analysing Deep Reinforcement Learning Agents Trained with Domain Randomisation
• Deep Reinforcement Learning for Human-Like Driving Policies in Collision Avoidance Tasks of Self-Driving Cars
• Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning
• Generalization in Transfer Learning
• Quantifying the effects of environment and population diversity in multi-agent reinforcement learning
• Distributionally Robust Reinforcement Learning
• Observational Overfitting in Reinforcement Learning
• Strategies for Using Proximal Policy Optimization in Mobile Puzzle Games
• Regularization Matters in Policy Optimization
• Generalization in Reinforcement Learning by Soft Data Augmentation
• MixStyle Neural Networks for Domain Generalization and Adaptation
• Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learning
• Neuropathic Pain Diagnosis Simulator for Causal Discovery Algorithm Evaluation
• Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves
• SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies
• Using a thousand optimization tasks to learn hyperparameter search strategies
• Counterfactual Data Augmentation using Locally Factored Dynamics
• Interactive Language Learning by Question Answering
• Interference and Generalization in Temporal Difference Learning
• On the Measure of Intelligence
• Generalizing from a few environments in safety-critical reinforcement learning
• One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL
• Rotation, Translation, and Cropping for Zero-Shot Generalization
• An Empirical Study on Hyperparameters and their Interdependence for RL Generalization
• Intervention Design for Effective Sim2Real Transfer
• Measuring Visual Generalization in Continuous Control from Pixels
• Reinforcement Learning Generalization with Surprise Minimization
• The MAGICAL Benchmark for Robust Imitation
• Uncertainty-sensitive Learning and Planning with Ensembles
• Toybox: A Suite of Environments for Experimental Evaluation of Deep Reinforcement Learning
• A Convex Neural Network Solver for DCOPF with Generalization Guarantees
• Inference of Affordances and Active Motor Control in Simulated Agents
• Reasoning and Generalization in RL: A Tool Use Perspective
• Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning
• Deep RL With Information Constrained Policies: Generalization in Continuous Control
• Unsupervised Emergence of Egocentric Spatial Structure from Sensorimotor Prediction
• Decoupling Value and Policy for Generalization in Reinforcement Learning
• Counterfactual Policy Evaluation for Decision-Making in Autonomous Driving
• Evolutionary Computation and AI Safety: Research Problems Impeding Routine and Safe Real-world Application of Evolution
• A Brief Look at Generalization in Visual Meta-Reinforcement Learning
• Group Equivariant Deep Reinforcement Learning
• Generalization to Novel Objects using Prior Relational Knowledge
• Perceptual Values from Observation
• Towards a practical measure of interference for reinforcement learning
• Meta Arcade: A Configurable Environment Suite for Meta-Learning
• Instance based Generalization in Reinforcement Learning
• Rogue-Gym: A New Challenge for Generalization in Reinforcement Learning
• An investigation of model-free planning
• Transfer Reinforcement Learning across Homotopy Classes
• Relational Mimic for Visual Adversarial Imitation Learning
• Wield: Systematic Reinforcement Learning With Progressive Randomization
• Follow the Neurally-Perturbed Leader for Adversarial Training
• Extrapolation in Gridworld Markov-Decision Processes
• Enhanced Scene Specificity with Sparse Dynamic Value Estimation