A Survey of Zero-shot Generalisation in Deep Reinforcement Learning

Abstract

The study of zero-shot generalisation (ZSG) in deep Reinforcement Learning (RL) aims to produce RL algorithms whose policies generalise well to novel unseen situations at deployment time, avoiding overfitting to their training environments. Tackling this is vital if we are to deploy reinforcement learning algorithms in real world scenarios, where the environment will be diverse, dynamic and unpredictable. This survey is an overview of this nascent field. We rely on a unifying formalism and terminology for discussing different ZSG problems, building upon previous works. We go on to categorise existing benchmarks for ZSG, as well as current methods for tackling these problems. Finally, we provide a critical discussion of the current state of the field, including recommendations for future work. Among other conclusions, we argue that taking a purely procedural content generation approach to benchmark design is not conducive to progress in ZSG, we suggest fast online adaptation and tackling RL-specific problems as some areas for future work on methods for ZSG, and we recommend building benchmarks in underexplored problem settings such as offline RL ZSG and reward-function variation.

arXiv commentJAIR version. Added formal definitions of ZSPT and related concepts, JAIR formatting, other small rewrites; https://www.jair.org/index.php/jair/article/view/14174

References in corpus (124)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• mixup: Beyond Empirical Risk Minimization
• Improved Regularization of Convolutional Neural Networks with Cutout
• Domain Generalization: A Survey
• Dota 2 with Large Scale Deep Reinforcement Learning
• A Simple Neural Attentive Meta-Learner
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Solving Rubik's Cube with a Robot Hand
• OpenAI Gym
• Conservative Q-Learning for Offline Reinforcement Learning
• DeepMind Control Suite
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Autonomous Agents Modelling Other Agents: A Comprehensive Survey and Open Problems
• Robust Adversarial Reinforcement Learning
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• A Study on Overfitting in Deep Reinforcement Learning
• Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey
• Maximum a Posteriori Policy Optimisation
• Imagination-Augmented Agents for Deep Reinforcement Learning
• Quantifying Generalization in Reinforcement Learning
• Leveraging Procedural Generation to Benchmark Reinforcement Learning
• robosuite: A Modular Simulation Framework and Benchmark for Robot Learning
• Relational Deep Reinforcement Learning
• Transfer Learning in Deep Reinforcement Learning: A Survey
• Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physics
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• DARLA: Improving Zero-Shot Transfer in Reinforcement Learning
• Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions
• Assessing Generalization in Deep Reinforcement Learning
• Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning
• Diversity is All You Need: Learning Skills without a Reward Function
• Generalization and Regularization in DQN
• A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning
• Learning Invariant Representations for Reinforcement Learning without Reconstruction
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning
• VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning
• Self-Supervised Policy Adaptation during Deployment
• Multi-Task Reinforcement Learning with Soft Modularization
• Transfer Learning for Related Reinforcement Learning Tasks via Image-to-Image Translation
• Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneck
• Open-Ended Learning Leads to Generally Capable Agents
• Measuring Compositional Generalization: A Comprehensive Method on Realistic Data
• Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts?
• Environmental drivers of systematicity and generalization in a situated agent
• Automatic Data Augmentation for Generalization in Deep Reinforcement Learning
• Provably efficient RL with Rich Observations via Latent State Decoding
• An empirical investigation of the challenges of real-world reinforcement learning
• Phasic Policy Gradient
• Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL
• Hidden Parameter Markov Decision Processes: A Semiparametric Regression Approach for Discovering Latent Task Parametrizations
• Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learning
• The NetHack Learning Environment
• Improving Generalization in Reinforcement Learning with Mixture Regularization
• Near-optimal Reinforcement Learning in Factored MDPs
• iGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenes
• Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions
• Preparing for the Unknown: Learning a Universal Policy with Online System Identification
• Robust Reinforcement Learning for Continuous Control with Model Misspecification
• Invariant Causal Prediction for Block MDPs
• Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation
• Wasserstein Robust Reinforcement Learning
• Investigating Generalisation in Continuous Deep Reinforcement Learning
• Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text
• Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning?
• CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning
• Natural Environment Benchmarks for Reinforcement Learning
• Towards Continual Reinforcement Learning: A Review and Perspectives
• Generalization of Reinforcement Learners with Working and Episodic Memory
• Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning
• Online and Offline Reinforcement Learning by Planning with a Learned Model
• A Survey of Explainable Reinforcement Learning
• Interactive Fiction Games: A Colossal Adventure
• Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design
• Dynamics Generalization via Information Bottleneck in Deep Reinforcement Learning
• Observational Overfitting in Reinforcement Learning
• Generalization in Reinforcement Learning by Soft Data Augmentation
• RTFM: Generalising to Novel Environment Dynamics via Reading
• SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies
• Invariant Policy Optimization: Towards Stronger Generalization in Reinforcement Learning
• The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixels
• MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Research
• Procedural Content Generation: Better Benchmarks for Transfer Reinforcement Learning
• Generalization to New Actions in Reinforcement Learning
• Interference and Generalization in Temporal Difference Learning
• WordCraft: An Environment for Benchmarking Commonsense Agents
• Transient Non-Stationarity and Generalisation in Deep Reinforcement Learning
• Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability
• The Sensory Neuron as a Transformer: Permutation-Invariant Neural Networks for Reinforcement Learning
• Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environment
• CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents
• RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning
• HALMA: Humanlike Abstraction Learning Meets Affordance in Rapid Problem Solving
• Improving Generalization of Reinforcement Learning with Minimax Distributional Soft Actor-Critic
• Reinforcement Learning Generalization with Surprise Minimization
• Measuring Visual Generalization in Continuous Control from Pixels
• Cross-Trajectory Representation Learning for Zero-Shot Generalization in RL
• Trajectory-wise Multiple Choice Learning for Dynamics Generalization in Reinforcement Learning
• Ask Your Humans: Using Human Instructions to Improve Generalization in Reinforcement Learning
• Toybox: A Suite of Environments for Experimental Evaluation of Deep Reinforcement Learning
• AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learning
• Generalization of Reinforcement Learning with Policy-Aware Adversarial Data Augmentation
• When Is Generalizable Reinforcement Learning Tractable?
• Domain Adversarial Reinforcement Learning
• Robust Domain Randomised Reinforcement Learning through Peer-to-Peer Distillation
• Reinforcement Learning with Videos: Combining Offline Observations with Interaction
• Reasoning and Generalization in RL: A Tool Use Perspective
• A Survey of Exploration Methods in Reinforcement Learning
• Decoupling Value and Policy for Generalization in Reinforcement Learning
• Unsupervised Visual Attention and Invariance for Reinforcement Learning
• Procedural Generalization by Planning with Self-Supervised World Models
• Neuro-algorithmic Policies enable Fast Combinatorial Generalization
• Block Contextual MDPs for Continual Learning
• Language Conditioned Imitation Learning over Unstructured Data
• DRIBO: Robust Deep Reinforcement Learning via Multi-View Information Bottleneck
• Learning Invariances for Policy Generalization
• CARL: A Benchmark for Contextual and Adaptive Reinforcement Learning
• Exploration in Approximate Hyper-State Space for Meta Reinforcement Learning
• Online Sparse Reinforcement Learning
• Sparse Attention Guided Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement Learning
• The Principle of Unchanged Optimality in Reinforcement Learning Generalization
• Instance based Generalization in Reinforcement Learning
• Rogue-Gym: A New Challenge for Generalization in Reinforcement Learning

Cited by in corpus (12)

• Domain Generalization: A Survey
• State-of-the-art generalisation research in NLP: A taxonomy and review
• Generalization in Deep Reinforcement Learning for Robotic Navigation by Reward Shaping
• A Tutorial on Meta-Reinforcement Learning
• Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challenges
• Maximum diffusion reinforcement learning
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• GreenLight-Gym: Reinforcement learning benchmark environment for control of greenhouse production systems
• Sim-to-Real Transfer via a Style-Identified Cycle Consistent Generative Adversarial Network: Zero-Shot Deployment on Robotic Manipulators through Visual Domain Adaptation
• The RL Perceptron: Generalisation Dynamics of Policy Learning in High Dimensions
• Improving greenhouse fruit-production control by integrating reinforcement learning into short-horizon model predictive control
• Dual Variational Autoencoders for Efficient Sim-to-Real Transfer in Low-Cost Robotic Navigation