Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels

Abstract

We propose a simple data augmentation technique that can be applied to standard model-free reinforcement learning algorithms, enabling robust learning directly from pixels without the need for auxiliary losses or pre-training. The approach leverages input perturbations commonly used in computer vision tasks to regularize the value function. Existing model-free approaches, such as Soft Actor-Critic (SAC), are not able to train deep networks effectively from image pixels. However, the addition of our augmentation method dramatically improves SAC's performance, enabling it to reach state-of-the-art performance on the DeepMind control suite, surpassing model-based (Dreamer, PlaNet, and SLAC) methods and recently proposed contrastive learning (CURL). Our approach can be combined with any model-free reinforcement learning algorithm, requiring only minor modifications. An implementation can be found at https://sites.google.com/view/data-regularized-q.

References in corpus (34)

• Adam: A Method for Stochastic Optimization
• Continuous control with deep reinforcement learning
• A Simple Framework for Contrastive Learning of Visual Representations
• Improving neural networks by preventing co-adaptation of feature detectors
• Natural Language Processing (almost) from Scratch
• Playing Atari with Deep Reinforcement Learning
• Improved Regularization of Convolutional Neural Networks with Cutout
• Soft Actor-Critic Algorithms and Applications
• Semi-Supervised Learning with Deep Generative Models
• Momentum Contrast for Unsupervised Visual Representation Learning
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• DeepMind Control Suite
• Model-Based Reinforcement Learning for Atari
• Deep Reinforcement Learning that Matters
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Reinforcement Learning with Unsupervised Auxiliary Tasks
• Reinforcement Learning with Augmented Data
• Maximum a Posteriori Policy Optimisation
• High-Performance Neural Networks for Visual Object Classification
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• Quantifying Generalization in Reinforcement Learning
• Leveraging Procedural Generation to Benchmark Reinforcement Learning
• Dream to Control: Learning Behaviors by Latent Imagination
• When to use parametric models in reinforcement learning?
• Generalization and Regularization in DQN
• Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model
• Robot Learning in Homes: Improving Generalization and Reducing Dataset Bias
• Understanding the impact of entropy on policy optimization
• Regularization Matters in Policy Optimization
• Learning Actionable Representations from Visual Observations
• Robust Visual Domain Randomization for Reinforcement Learning
• Manifold Regularization for Kernelized LSTD

Cited by in corpus (73)

• Reinforcement Learning with Augmented Data
• CURL: Contrastive Unsupervised Representations for Reinforcement Learning
• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
• SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation
• Decoupling Representation Learning from Reinforcement Learning
• Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model
• Self-Supervised Policy Adaptation during Deployment
• Automatic Data Augmentation for Generalization in Deep Reinforcement Learning
• Data-Efficient Reinforcement Learning with Self-Predictive Representations
• Meta-Learning Requires Meta-Augmentation
• SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning
• MDP Homomorphic Networks: Group Symmetries in Reinforcement Learning
• Domain Generalization with MixStyle
• Reinforcement Learning with Prototypical Representations
• Representation Learning via Invariant Causal Mechanisms
• Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning
• Mastering Atari with Discrete World Models
• Predictive Information Accelerates Learning in RL
• Sample-efficient Reinforcement Learning Representation Learning with Curiosity Contrastive Forward Dynamics Model
• Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics
• Visual Imitation Made Easy
• Offline Reinforcement Learning from Images with Latent Space Models
• On Data-Augmentation and Consistency-Based Semi-Supervised Learning
• Generalization in Reinforcement Learning by Soft Data Augmentation
• S3K: Self-Supervised Semantic Keypoints for Robotic Manipulation via Multi-View Consistency
• Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation
• Learning Visual Robotic Control Efficiently with Contrastive Pre-training and Data Augmentation
• SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies
• Recurrent Off-policy Baselines for Memory-based Continuous Control
• The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixels
• D2RL: Deep Dense Architectures in Reinforcement Learning
• CARLA Real Traffic Scenarios -- novel training ground and benchmark for autonomous driving
• Behavior From the Void: Unsupervised Active Pre-Training
• DreamerPro: Reconstruction-Free Model-Based Reinforcement Learning with Prototypical Representations
• Low-Precision Reinforcement Learning: Running Soft Actor-Critic in Half Precision
• MADE: Exploration via Maximizing Deviation from Explored Regions
• Counterfactual Data Augmentation using Locally Factored Dynamics
• ESP: Exploiting Symmetry Prior for Multi-Agent Reinforcement Learning
• Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability
• Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstruction
• A review of mobile robot motion planning methods: from classical motion planning workflows to reinforcement learning-based architectures
• Rotation, Translation, and Cropping for Zero-Shot Generalization
• Intervention Design for Effective Sim2Real Transfer
• Measuring Visual Generalization in Continuous Control from Pixels
• Provable Representation Learning for Imitation with Contrastive Fourier Features
• Exploratory State Representation Learning
• Transformers for One-Shot Visual Imitation
• Hindsight Expectation Maximization for Goal-conditioned Reinforcement Learning
• Domain Adversarial Reinforcement Learning
• Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddings
• S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning
• Visual Adversarial Imitation Learning using Variational Models
• Neuro-algorithmic Policies enable Fast Combinatorial Generalization
• Decoupling Value and Policy for Generalization in Reinforcement Learning
• Equivariant $Q$ Learning in Spatial Action Spaces
• Prioritized Level Replay
• Continuous Transition: Improving Sample Efficiency for Continuous Control Problems via MixUp
• Reinforcement Learning with Feedback Graphs
• Direct Differentiable Augmentation Search
• Pre-training of Deep RL Agents for Improved Learning under Domain Randomization
• Towards Deeper Deep Reinforcement Learning with Spectral Normalization
• Learning State Representations via Retracing in Reinforcement Learning
• Learning Temporally-Consistent Representations for Data-Efficient Reinforcement Learning
• Hypernetwork-Based Augmentation
• Reinforcement Learning with Latent Flow
• Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via Latent Model Ensembles
• Learning Representations for Pixel-based Control: What Matters and Why?
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• Is High Variance Unavoidable in RL? A Case Study in Continuous Control
• Utilizing Skipped Frames in Action Repeats via Pseudo-Actions
• Vision-Guided Quadrupedal Locomotion in the Wild with Multi-Modal Delay Randomization
• A Geometric Perspective on Self-Supervised Policy Adaptation