Mid-Level Visual Representations Improve Generalization and Sample Efficiency for Learning Visuomotor Policies

Abstract

How much does having visual priors about the world (e.g. the fact that the world is 3D) assist in learning to perform downstream motor tasks (e.g. delivering a package)? We study this question by integrating a generic perceptual skill set (e.g. a distance estimator, an edge detector, etc.) within a reinforcement learning framework--see Figure 1. This skill set (hereafter mid-level perception) provides the policy with a more processed state of the world compared to raw images. We find that using a mid-level perception confers significant advantages over training end-to-end from scratch (i.e. not leveraging priors) in navigation-oriented tasks. Agents are able to generalize to situations where the from-scratch approach fails and training becomes significantly more sample efficient. However, we show that realizing these gains requires careful selection of the mid-level perceptual skills. Therefore, we refine our findings into an efficient max-coverage feature set that can be adopted in lieu of raw images. We perform our study in completely separate buildings for training and testing and compare against visually blind baseline policies and state-of-the-art feature learning methods.

arXiv commentSee project website, demos, and code at http://perceptual.actor

References in corpus (20)

• Representation Learning with Contrastive Predictive Coding
• Playing Atari with Deep Reinforcement Learning
• Learning Transferable Features with Deep Adaptation Networks
• Deep Domain Confusion: Maximizing for Domain Invariance
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• Analysis of Thompson Sampling for the multi-armed bandit problem
• Semi-Supervised Learning with Generative Adversarial Networks
• On Evaluation of Embodied Navigation Agents
• Adversarial Discriminative Domain Adaptation
• YOLO9000: Better, Faster, Stronger
• Deep CORAL: Correlation Alignment for Deep Domain Adaptation
• Learning to Poke by Poking: Experiential Learning of Intuitive Physics
• DARLA: Improving Zero-Shot Transfer in Reinforcement Learning
• Learning to Act by Predicting the Future
• One-Shot Hierarchical Imitation Learning of Compound Visuomotor Tasks
• Moment Matching for Multi-Source Domain Adaptation
• Learning Exploration Policies for Navigation
• S-RL Toolbox: Environments, Datasets and Evaluation Metrics for State Representation Learning
• Generalization through Simulation: Integrating Simulated and Real Data into Deep Reinforcement Learning for Vision-Based Autonomous Flight
• Learning 3D Human Dynamics from Video

Cited by in corpus (19)

• On-edge Multi-task Transfer Learning: Model and Practice with Data-driven Task Allocation
• Embodied Visual Navigation with Automatic Curriculum Learning in Real Environments
• Learning to Set Waypoints for Audio-Visual Navigation
• Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers
• On the Sample Complexity of Stability Constrained Imitation Learning
• Auxiliary Tasks and Exploration Enable ObjectNav
• Where are the Keys? -- Learning Object-Centric Navigation Policies on Semantic Maps with Graph Convolutional Networks
• Auxiliary Tasks Speed Up Learning PointGoal Navigation
• Object Finding in Cluttered Scenes Using Interactive Perception
• Visuomotor Understanding for Representation Learning of Driving Scenes
• Learning Continuous Environment Fields via Implicit Functions
• MELD: Meta-Reinforcement Learning from Images via Latent State Models
• Feedback from Pixels: Output Regulation via Learning-Based Scene View Synthesis
• A Simple Approach to Continual Learning by Transferring Skill Parameters
• MaAST: Map Attention with Semantic Transformersfor Efficient Visual Navigation
• The Pitfall of More Powerful Autoencoders in Lidar-Based Navigation
• A Few Shot Adaptation of Visual Navigation Skills to New Observations using Meta-Learning
• Vision-Guided Quadrupedal Locomotion in the Wild with Multi-Modal Delay Randomization
• Learning to See before Learning to Act: Visual Pre-training for Manipulation