Learning Visual Robotic Control Efficiently with Contrastive Pre-training and Data Augmentation

Abstract

Recent advances in unsupervised representation learning significantly improved the sample efficiency of training Reinforcement Learning policies in simulated environments. However, similar gains have not yet been seen for real-robot reinforcement learning. In this work, we focus on enabling data-efficient real-robot learning from pixels. We present Contrastive Pre-training and Data Augmentation for Efficient Robotic Learning (CoDER), a method that utilizes data augmentation and unsupervised learning to achieve sample-efficient training of real-robot arm policies from sparse rewards. While contrastive pre-training, data augmentation, demonstrations, and reinforcement learning are alone insufficient for efficient learning, our main contribution is showing that the combination of these disparate techniques results in a simple yet data-efficient method. We show that, given only 10 demonstrations, a single robotic arm can learn sparse-reward manipulation policies from pixels, such as reaching, picking, moving, pulling a large object, flipping a switch, and opening a drawer in just 30 minutes of mean real-world training time. We include videos and code on the project website: https://sites.google.com/view/efficient-robotic-manipulation/home

References in corpus (9)

• DeepMind Control Suite
• One-Shot Visual Imitation Learning via Meta-Learning
• Reinforcement Learning with Augmented Data
• Agent57: Outperforming the Atari Human Benchmark
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• Learning Predictive Representations for Deformable Objects Using Contrastive Estimation
• Decoupling Representation Learning from Reinforcement Learning
• Visual Imitation Made Easy
• Online Object Representations with Contrastive Learning

Cited by in corpus (9)

• What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
• Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning
• Reinforcement Learning with Prototypical Representations
• Low-Precision Reinforcement Learning: Running Soft Actor-Critic in Half Precision
• Which Mutual-Information Representation Learning Objectives are Sufficient for Control?
• Learning Multi-Stage Tasks with One Demonstration via Self-Replay
• Equivariant $Q$ Learning in Spatial Action Spaces
• Is High Variance Unavoidable in RL? A Case Study in Continuous Control
• Towards a Sample Efficient Reinforcement Learning Pipeline for Vision Based Robotics