Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning

Abstract

We present DrQ-v2, a model-free reinforcement learning (RL) algorithm for visual continuous control. DrQ-v2 builds on DrQ, an off-policy actor-critic approach that uses data augmentation to learn directly from pixels. We introduce several improvements that yield state-of-the-art results on the DeepMind Control Suite. Notably, DrQ-v2 is able to solve complex humanoid locomotion tasks directly from pixel observations, previously unattained by model-free RL. DrQ-v2 is conceptually simple, easy to implement, and provides significantly better computational footprint compared to prior work, with the majority of tasks taking just 8 hours to train on a single GPU. Finally, we publicly release DrQ-v2's implementation to provide RL practitioners with a strong and computationally efficient baseline.

References in corpus (16)

• Playing Atari with Deep Reinforcement Learning
• Soft Actor-Critic Algorithms and Applications
• DeepMind Control Suite
• Reinforcement Learning with Augmented Data
• Dream to Control: Learning Behaviors by Latent Imagination
• Safe and Efficient Off-Policy Reinforcement Learning
• Learning Predictive Representations for Deformable Objects Using Contrastive Estimation
• Decoupling Representation Learning from Reinforcement Learning
• Automatic Data Augmentation for Generalization in Deep Reinforcement Learning
• Data-Efficient Reinforcement Learning with Self-Predictive Representations
• Reinforcement Learning with Prototypical Representations
• Mastering Atari with Discrete World Models
• Visual Imitation Made Easy
• Robust Policies via Mid-Level Visual Representations: An Experimental Study in Manipulation and Navigation
• On the model-based stochastic value gradient for continuous reinforcement learning
• Learning Visual Robotic Control Efficiently with Contrastive Pre-training and Data Augmentation

Cited by in corpus (10)

• RLAD: Reinforcement Learning from Pixels for Autonomous Driving in Urban Environments
• CCLF: A Contrastive-Curiosity-Driven Learning Framework for Sample-Efficient Reinforcement Learning
• Counterfactual experience augmented off-policy reinforcement learning
• CROP: Towards Distributional-Shift Robust Reinforcement Learning using Compact Reshaped Observation Processing
• Towards Deeper Deep Reinforcement Learning with Spectral Normalization
• Learning State Representations via Retracing in Reinforcement Learning
• Learning Pessimism for Robust and Efficient Off-Policy Reinforcement Learning
• A Champion-level Vision-based Reinforcement Learning Agent for Competitive Racing in Gran Turismo 7
• Maximum Entropy Model-based Reinforcement Learning
• Is High Variance Unavoidable in RL? A Case Study in Continuous Control