Robust Policies via Mid-Level Visual Representations: An Experimental Study in Manipulation and Navigation

Abstract

Vision-based robotics often separates the control loop into one module for perception and a separate module for control. It is possible to train the whole system end-to-end (e.g. with deep RL), but doing it "from scratch" comes with a high sample complexity cost and the final result is often brittle, failing unexpectedly if the test environment differs from that of training. We study the effects of using mid-level visual representations (features learned asynchronously for traditional computer vision objectives), as a generic and easy-to-decode perceptual state in an end-to-end RL framework. Mid-level representations encode invariances about the world, and we show that they aid generalization, improve sample complexity, and lead to a higher final performance. Compared to other approaches for incorporating invariances, such as domain randomization, asynchronously trained mid-level representations scale better: both to harder problems and to larger domain shifts. In practice, this means that mid-level representations could be used to successfully train policies for tasks where domain randomization and learning-from-scratch failed. We report results on both manipulation and navigation tasks, and for navigation include zero-shot sim-to-real experiments on real robots.

Extended version of CoRL 2020 camera ready. Supplementary released separately

References in corpus (24)

• A Simple Framework for Contrastive Learning of Visual Representations
• Playing Atari with Deep Reinforcement Learning
• Deep Residual Learning for Image Recognition
• Bootstrap your own latent: A new approach to self-supervised Learning
• Addressing Function Approximation Error in Actor-Critic Methods
• End-to-End Training of Deep Visuomotor Policies
• Momentum Contrast for Unsupervised Visual Representation Learning
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
• Solving Rubik's Cube with a Robot Hand
• Learning multiple visual domains with residual adapters
• On Evaluation of Embodied Navigation Agents
• Hindsight Experience Replay
• Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
• Learning to Poke by Poking: Experiential Learning of Intuitive Physics
• PyRep: Bringing V-REP to Deep Robot Learning
• Learning Predictive Representations for Deformable Objects Using Contrastive Estimation
• Test-Time Training with Self-Supervision for Generalization under Distribution Shifts
• Learning to Act by Predicting the Future
• Self-Supervised Policy Adaptation during Deployment
• Robot Learning in Homes: Improving Generalization and Reducing Dataset Bias
• S-RL Toolbox: Environments, Datasets and Evaluation Metrics for State Representation Learning
• Learning to Manipulate Deformable Objects without Demonstrations
• Learning to Navigate Using Mid-Level Visual Priors

Cited by in corpus (7)

• Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning
• Reinforcement Learning with Prototypical Representations
• Learning Continuous Environment Fields via Implicit Functions
• Learning Cross-Domain Correspondence for Control with Dynamics Cycle-Consistency
• Dual-Arm Adversarial Robot Learning
• Investigation of Factorized Optical Flows as Mid-Level Representations
• Few-Shot Image-to-Semantics Translation for Policy Transfer in Reinforcement Learning