Automatic Data Augmentation for Generalization in Deep Reinforcement Learning

Abstract

Deep reinforcement learning (RL) agents often fail to generalize to unseen scenarios, even when they are trained on many instances of semantically similar environments. Data augmentation has recently been shown to improve the sample efficiency and generalization of RL agents. However, different tasks tend to benefit from different kinds of data augmentation. In this paper, we compare three approaches for automatically finding an appropriate augmentation. These are combined with two novel regularization terms for the policy and value function, required to make the use of data augmentation theoretically sound for certain actor-critic algorithms. We evaluate our methods on the Procgen benchmark which consists of 16 procedurally-generated environments and show that it improves test performance by ~40% relative to standard RL algorithms. Our agent outperforms other baselines specifically designed to improve generalization in RL. In addition, we show that our agent learns policies and representations that are more robust to changes in the environment that do not affect the agent, such as the background. Our implementation is available at https://github.com/rraileanu/auto-drac.

References in corpus (38)

• A Simple Framework for Contrastive Learning of Visual Representations
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Playing Atari with Deep Reinforcement Learning
• Trust Region Policy Optimization
• The Kinetics Human Action Video Dataset
• Improved Regularization of Convolutional Neural Networks with Cutout
• Momentum Contrast for Unsupervised Visual Representation Learning
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• DeepMind Control Suite
• Learning to reinforcement learn
• Reinforcement Learning with Augmented Data
• High-Performance Neural Networks for Visual Object Classification
• Sample Efficient Actor-Critic with Experience Replay
• Quantifying Generalization in Reinforcement Learning
• Leveraging Procedural Generation to Benchmark Reinforcement Learning
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation
• Assessing Generalization in Deep Reinforcement Learning
• Generalization and Regularization in DQN
• Generalized Inner Loop Meta-Learning
• A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning
• Gotta Learn Fast: A New Benchmark for Generalization in RL
• Learning Invariant Representations for Reinforcement Learning without Reconstruction
• Decoupling Representation Learning from Reinforcement Learning
• Transfer Learning for Related Reinforcement Learning Tasks via Image-to-Image Translation
• Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneck
• Improving Generalization in Reinforcement Learning with Mixture Regularization
• The NetHack Learning Environment
• DeepMind Lab
• Invariant Causal Prediction for Block MDPs
• RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments
• Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning
• Selecting Data Augmentation for Simulating Interventions
• Invariant Policy Optimization: Towards Stronger Generalization in Reinforcement Learning
• Rotation, Translation, and Cropping for Zero-Shot Generalization
• Measuring Visual Generalization in Continuous Control from Pixels
• Prioritized Level Replay
• Automatic Data Augmentation by Learning the Deterministic Policy

Cited by in corpus (20)

• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• Review: Deep Learning in Electron Microscopy
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning
• Deep Reinforcement Learning at the Edge of the Statistical Precipice
• Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation
• Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning
• Generalization in Reinforcement Learning by Soft Data Augmentation
• Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation
• Learning Task-relevant Representations for Generalization via Characteristic Functions of Reward Sequence Distributions
• SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies
• CCLF: A Contrastive-Curiosity-Driven Learning Framework for Sample-Efficient Reinforcement Learning
• Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability
• Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environment
• Counterfactual experience augmented off-policy reinforcement learning
• Decoupling Value and Policy for Generalization in Reinforcement Learning
• Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixture
• DRIBO: Robust Deep Reinforcement Learning via Multi-View Information Bottleneck
• Continuous Transition: Improving Sample Efficiency for Continuous Control Problems via MixUp
• Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRL