Assessing Generalization in Deep Reinforcement Learning

Abstract

Deep reinforcement learning (RL) has achieved breakthrough results on many tasks, but agents often fail to generalize beyond the environment they were trained in. As a result, deep RL algorithms that promote generalization are receiving increasing attention. However, works in this area use a wide variety of tasks and experimental setups for evaluation. The literature lacks a controlled assessment of the merits of different generalization schemes. Our aim is to catalyze community-wide progress on generalization in deep RL. To this end, we present a benchmark and experimental protocol, and conduct a systematic empirical study. Our framework contains a diverse set of environments, our methodology covers both in-distribution and out-of-distribution generalization, and our evaluation includes deep RL algorithms that specifically tackle generalization. Our key finding is that `vanilla' deep RL algorithms generalize better than specialized schemes that were proposed specifically to tackle generalization.

arXiv comment17 pages, 6 figures

References in corpus (19)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• A Brief Survey of Deep Reinforcement Learning
• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
• An Overview of Multi-Task Learning in Deep Neural Networks
• Benchmarking Deep Reinforcement Learning for Continuous Control
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Deep Learning: A Critical Appraisal
• Learning to reinforcement learn
• Quantifying Generalization in Reinforcement Learning
• Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physics
• Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments
• EPOpt: Learning Robust Neural Network Policies Using Model Ensembles
• AI Safety Gridworlds
• Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation
• Meta Reinforcement Learning with Latent Variable Gaussian Processes
• Learning to Learn: Meta-Critic Networks for Sample Efficient Learning
• A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning
• Gotta Learn Fast: A New Benchmark for Generalization in RL
• Reinforcement Learning under Model Mismatch

Cited by in corpus (54)

• Quantifying Generalization in Reinforcement Learning
• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• Leveraging Procedural Generation to Benchmark Reinforcement Learning
• Self-Supervised Policy Adaptation during Deployment
• Characterizing Attacks on Deep Reinforcement Learning
• Automatic Data Augmentation for Generalization in Deep Reinforcement Learning
• Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learning
• TAP-Net: Transport-and-Pack using Reinforcement Learning
• Learning to Locomote: Understanding How Environment Design Matters for Deep Reinforcement Learning
• Towards Joint Learning of Optimal MAC Signaling and Wireless Channel Access
• Wasserstein Robust Reinforcement Learning
• Investigating Generalisation in Continuous Deep Reinforcement Learning
• Active Domain Randomization
• RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments
• CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning
• Generalizable control for multiparameter quantum metrology
• Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning
• Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learning
• Analysing Deep Reinforcement Learning Agents Trained with Domain Randomisation
• Distributionally Robust Reinforcement Learning
• Regularization Matters in Policy Optimization
• Dynamics Generalization via Information Bottleneck in Deep Reinforcement Learning
• Observational Overfitting in Reinforcement Learning
• Strategies for Using Proximal Policy Optimization in Mobile Puzzle Games
• Generalization in Reinforcement Learning by Soft Data Augmentation
• From Seeing to Moving: A Survey on Learning for Visual Indoor Navigation (VIN)
• SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies
• Self-Attentional Credit Assignment for Transfer in Reinforcement Learning
• Reinforcement Learning with Dual-Observation for General Video Game Playing
• Interference and Generalization in Temporal Difference Learning
• Modeling and Optimization Trade-off in Meta-learning
• Improving Generalization of Reinforcement Learning with Minimax Distributional Soft Actor-Critic
• Trajectory-wise Multiple Choice Learning for Dynamics Generalization in Reinforcement Learning
• Robust Visual Domain Randomization for Reinforcement Learning
• When Is Generalizable Reinforcement Learning Tractable?
• Investigating Simple Object Representations in Model-Free Deep Reinforcement Learning
• Deep RL With Information Constrained Policies: Generalization in Continuous Control
• MRAC-RL: A Framework for On-Line Policy Adaptation Under Parametric Model Uncertainty
• Reasoning and Generalization in RL: A Tool Use Perspective
• Robust Domain Randomised Reinforcement Learning through Peer-to-Peer Distillation
• Unsupervised Emergence of Egocentric Spatial Structure from Sensorimotor Prediction
• Joint Attention for Multi-Agent Coordination and Social Learning
• Decoupling Value and Policy for Generalization in Reinforcement Learning
• AutoSoC: Automating Algorithm-SOC Co-design for Aerial Robots
• Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRL
• Fully Distributed Actor-Critic Architecture for Multitask Deep Reinforcement Learning
• Learning Domain Randomization Distributions for Training Robust Locomotion Policies
• Towards a practical measure of interference for reinforcement learning
• Generalization to Novel Objects using Prior Relational Knowledge
• Correcting Momentum in Temporal Difference Learning
• Online Algorithms and Policies Using Adaptive and Machine Learning Approaches
• Zero-shot Policy Learning with Spatial Temporal RewardDecomposition on Contingency-aware Observation
• Extrapolation in Gridworld Markov-Decision Processes
• Enhanced Scene Specificity with Sparse Dynamic Value Estimation