Reinforcement Learning with Augmented Data

Abstract

Learning from visual observations is a fundamental yet challenging problem in Reinforcement Learning (RL). Although algorithmic advances combined with convolutional neural networks have proved to be a recipe for success, current methods are still lacking on two fronts: (a) data-efficiency of learning and (b) generalization to new environments. To this end, we present Reinforcement Learning with Augmented Data (RAD), a simple plug-and-play module that can enhance most RL algorithms. We perform the first extensive study of general data augmentations for RL on both pixel-based and state-based inputs, and introduce two new data augmentations - random translate and random amplitude scale. We show that augmentations such as random translate, crop, color jitter, patch cutout, random convolutions, and amplitude scale can enable simple RL algorithms to outperform complex state-of-the-art methods across common benchmarks. RAD sets a new state-of-the-art in terms of data-efficiency and final performance on the DeepMind Control Suite benchmark for pixel-based control as well as OpenAI Gym benchmark for state-based control. We further demonstrate that RAD significantly improves test-time generalization over existing methods on several OpenAI ProcGen benchmarks. Our RAD module and training code are available at https://www.github.com/MishaLaskin/rad.

arXiv commentNeurIPS 2020 camera-ready version. First two authors contributed equally, website: https://mishalaskin.github.io/rad code: https://github.com/MishaLaskin/rad and https://github.com/pokaxpoka/rad_procgen

References in corpus (21)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• The NumPy array: a structure for efficient numerical computation
• Continuous control with deep reinforcement learning
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Language Models are Few-Shot Learners
• Trust Region Policy Optimization
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• Unsupervised Data Augmentation for Consistency Training
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• cuDNN: Efficient Primitives for Deep Learning
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Solving Rubik's Cube with a Robot Hand
• DeepMind Control Suite
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Deep Reinforcement Learning that Matters
• Learning Latent Dynamics for Planning from Pixels
• Hindsight Experience Replay
• Model-Ensemble Trust-Region Policy Optimization
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• Exploring Model-based Planning with Policy Networks

Cited by in corpus (63)

• CURL: Contrastive Unsupervised Representations for Reinforcement Learning
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
• SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation
• Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning
• Decoupling Representation Learning from Reinforcement Learning
• Self-Supervised Policy Adaptation during Deployment
• Automatic Data Augmentation for Generalization in Deep Reinforcement Learning
• Meta-Learning Requires Meta-Augmentation
• Improving Generalization in Reinforcement Learning with Mixture Regularization
• Domain Generalization with MixStyle
• Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation
• Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning
• Predictive Information Accelerates Learning in RL
• Sample-efficient Reinforcement Learning Representation Learning with Curiosity Contrastive Forward Dynamics Model
• Visual Imitation Made Easy
• Regularization Matters in Policy Optimization
• Generalization in Reinforcement Learning by Soft Data Augmentation
• Offline Reinforcement Learning from Images with Latent Space Models
• S3K: Self-Supervised Semantic Keypoints for Robotic Manipulation via Multi-View Consistency
• Learning Visual Robotic Control Efficiently with Contrastive Pre-training and Data Augmentation
• Learning Task-relevant Representations for Generalization via Characteristic Functions of Reward Sequence Distributions
• SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies
• The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixels
• D2RL: Deep Dense Architectures in Reinforcement Learning
• CCLF: A Contrastive-Curiosity-Driven Learning Framework for Sample-Efficient Reinforcement Learning
• CARLA Real Traffic Scenarios -- novel training ground and benchmark for autonomous driving
• Reinforcement Learning with Dual-Observation for General Video Game Playing
• Behavior From the Void: Unsupervised Active Pre-Training
• Counterfactual Data Augmentation using Locally Factored Dynamics
• MADE: Exploration via Maximizing Deviation from Explored Regions
• ESP: Exploiting Symmetry Prior for Multi-Agent Reinforcement Learning
• A review of mobile robot motion planning methods: from classical motion planning workflows to reinforcement learning-based architectures
• Offline Meta Learning of Exploration
• Rotation, Translation, and Cropping for Zero-Shot Generalization
• Measuring Visual Generalization in Continuous Control from Pixels
• Transformers for One-Shot Visual Imitation
• Domain Adversarial Reinforcement Learning
• Generalization of Reinforcement Learning with Policy-Aware Adversarial Data Augmentation
• Robust Asymmetric Learning in POMDPs
• Attention-Privileged Reinforcement Learning
• S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning
• Domain Adaptation In Reinforcement Learning Via Latent Unified State Representation
• Equivariant $Q$ Learning in Spatial Action Spaces
• Auto-Tuned Sim-to-Real Transfer
• Decoupling Value and Policy for Generalization in Reinforcement Learning
• CROP: Towards Distributional-Shift Robust Reinforcement Learning using Compact Reshaped Observation Processing
• Neuro-algorithmic Policies enable Fast Combinatorial Generalization
• Unsupervised Visual Attention and Invariance for Reinforcement Learning
• Vision Transformer for Learning Driving Policies in Complex Multi-Agent Environments
• Pre-training of Deep RL Agents for Improved Learning under Domain Randomization
• Reinforcement Learning with Latent Flow
• Reinforcement Learning with Feedback Graphs
• Learning State Representations via Retracing in Reinforcement Learning
• Pretrained Encoders are All You Need
• Learning to Navigate Sidewalks in Outdoor Environments
• Towards Deeper Deep Reinforcement Learning with Spectral Normalization
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via Latent Model Ensembles
• Learning Fast and Precise Pixel-to-Torque Control
• Value-Based Reinforcement Learning for Continuous Control Robotic Manipulation in Multi-Task Sparse Reward Settings
• Safety Enhancement for Deep Reinforcement Learning in Autonomous Separation Assurance
• Is High Variance Unavoidable in RL? A Case Study in Continuous Control