Learning Visual Robotic Control Efficiently with Contrastive Pre-training and Data Augmentation

Abstract

Recent advances in unsupervised representation learning significantly improved the sample efficiency of training Reinforcement Learning policies in simulated environments. However, similar gains have not yet been seen for real-robot reinforcement learning. In this work, we focus on enabling data-efficient real-robot learning from pixels. We present Contrastive Pre-training and Data Augmentation for Efficient Robotic Learning (CoDER), a method that utilizes data augmentation and unsupervised learning to achieve sample-efficient training of real-robot arm policies from sparse rewards. While contrastive pre-training, data augmentation, demonstrations, and reinforcement learning are alone insufficient for efficient learning, our main contribution is showing that the combination of these disparate techniques results in a simple yet data-efficient method. We show that, given only 10 demonstrations, a single robotic arm can learn sparse-reward manipulation policies from pixels, such as reaching, picking, moving, pulling a large object, flipping a switch, and opening a drawer in just 30 minutes of mean real-world training time. We include videos and code on the project website: https://sites.google.com/view/efficient-robotic-manipulation/home

References in corpus (23)

• Continuous control with deep reinforcement learning
• Trust Region Policy Optimization
• Soft Actor-Critic Algorithms and Applications
• End-to-End Training of Deep Visuomotor Policies
• Dota 2 with Large Scale Deep Reinforcement Learning
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• DeepMind Control Suite
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Visual Reinforcement Learning with Imagined Goals
• One-Shot Visual Imitation Learning via Meta-Learning
• Learning Dexterous In-Hand Manipulation
• Reinforcement Learning with Augmented Data
• Agent57: Outperforming the Atari Human Benchmark
• Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• Learning Predictive Representations for Deformable Objects Using Contrastive Estimation
• Decoupling Representation Learning from Reinforcement Learning
• Robot Learning in Homes: Improving Generalization and Reducing Dataset Bias
• Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitation
• Visual Imitation Made Easy
• Online Object Representations with Contrastive Learning

Cited by in corpus (9)

• What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
• Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning
• Reinforcement Learning with Prototypical Representations
• Low-Precision Reinforcement Learning: Running Soft Actor-Critic in Half Precision
• Which Mutual-Information Representation Learning Objectives are Sufficient for Control?
• Learning Multi-Stage Tasks with One Demonstration via Self-Replay
• Equivariant $Q$ Learning in Spatial Action Spaces
• Is High Variance Unavoidable in RL? A Case Study in Continuous Control
• Towards a Sample Efficient Reinforcement Learning Pipeline for Vision Based Robotics