Visual Reinforcement Learning with Imagined Goals

Abstract

For an autonomous agent to fulfill a wide range of user-specified goals at test time, it must be able to learn broadly applicable and general-purpose skill repertoires. Furthermore, to provide the requisite level of generality, these skills must handle raw sensory input such as images. In this paper, we propose an algorithm that acquires such general-purpose skills by combining unsupervised representation learning and reinforcement learning of goal-conditioned policies. Since the particular goals that might be required at test-time are not known in advance, the agent performs a self-supervised "practice" phase where it imagines goals and attempts to achieve them. We learn a visual representation with three distinct purposes: sampling goals for self-supervised practice, providing a structured transformation of raw sensory inputs, and computing a reward signal for goal reaching. We also propose a retroactive goal relabeling scheme to further improve the sample-efficiency of our method. Our off-policy algorithm is efficient enough to learn policies that operate on raw image observations and goals for a real-world robotic system, and substantially outperforms prior techniques.

arXiv comment15 pages, NeurIPS 2018

References in corpus (15)

• Auto-Encoding Variational Bayes
• The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
• Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robots
• Action-Conditional Video Prediction using Deep Networks in Atari Games
• Stochastic Neural Networks for Hierarchical Reinforcement Learning
• Temporal Difference Models: Model-Free Deep RL for Model-Based Control
• Discovering Hidden Factors of Variation in Deep Networks
• Learning to Poke by Poking: Experiential Learning of Intuitive Physics
• DARLA: Improving Zero-Shot Transfer in Reinforcement Learning
• Self-Supervised Visual Planning with Temporal Skip Connections
• Universal Planning Networks
• Learning Visual Servoing with Deep Features and Fitted Q-Iteration
• Time-Contrastive Networks: Self-Supervised Learning from Video
• PVEs: Position-Velocity Encoders for Unsupervised Learning of Structured State Representations
• Deep Visual Foresight for Planning Robot Motion

Cited by in corpus (95)

• On the Opportunities and Risks of Foundation Models
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
• Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
• A survey on intrinsic motivation in reinforcement learning
• DALL-E-Bot: Introducing Web-Scale Diffusion Models to Robotics
• Efficient Exploration via State Marginal Matching
• Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model
• Self-Supervised Policy Adaptation during Deployment
• A Survey on Influence Maximization: From an ML-Based Combinatorial Optimization
• Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
• End-to-End Pixel-Based Deep Active Inference for Body Perception and Action
• Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration
• Disentangling Factors of Variation Using Few Labels
• Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learning
• Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation
• Reinforcement Learning for Robotic Manipulation using Simulated Locomotion Demonstrations
• MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale
• Automated curricula through setter-solver interactions
• Goal-conditioned Imitation Learning
• Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learning
• Discrete and continuous representations and processing in deep learning: Looking forward
• Self-supervised Learning of Image Embedding for Continuous Control
• Learning Robotic Navigation from Experience: Principles, Methods, and Recent Results
• Learning to Reach Goals via Iterated Supervised Learning
• Sample-efficient Reinforcement Learning Representation Learning with Curiosity Contrastive Forward Dynamics Model
• World Model as a Graph: Learning Latent Landmarks for Planning
• Learning Latent Plans from Play
• State Entropy Maximization with Random Encoders for Efficient Exploration
• PlanGAN: Model-based Planning With Sparse Rewards and Multiple Goals
• Broadly-Exploring, Local-Policy Trees for Long-Horizon Task Planning
• Learning Robotic Manipulation through Visual Planning and Acting
• Goal-Directed Design Agents: Integrating Visual Imitation with One-Step Lookahead Optimization for Generative Design
• Sparse Graphical Memory for Robust Planning
• Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills
• Offline Reinforcement Learning from Images with Latent Space Models
• Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generation
• A-SFS: Semi-supervised Feature Selection based on Multi-task Self-supervision
• Hindsight Goal Ranking on Replay Buffer for Sparse Reward Environment
• Grounding Language to Autonomously-Acquired Skills via Goal Generation
• Intrinsically Motivated Discovery of Diverse Patterns in Self-Organizing Systems
• Learning Visual Robotic Control Efficiently with Contrastive Pre-training and Data Augmentation
• Goal-Auxiliary Actor-Critic for 6D Robotic Grasping with Point Clouds
• Hierarchically Organized Latent Modules for Exploratory Search in Morphogenetic Systems
• Sub-policy Adaptation for Hierarchical Reinforcement Learning
• Time Reversal as Self-Supervision
• Hyperparameter Auto-tuning in Self-Supervised Robotic Learning
• Counterfactual Data Augmentation using Locally Factored Dynamics
• From Scratch to Sketch: Deep Decoupled Hierarchical Reinforcement Learning for Robotic Sketching Agent
• Self-supervised Learning of Distance Functions for Goal-Conditioned Reinforcement Learning
• Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks
• Hindsight Trust Region Policy Optimization
• Offline Meta-Reinforcement Learning with Online Self-Supervision
• On the Fairness of Disentangled Representations
• Weakly Supervised Disentangled Representation for Goal-conditioned Reinforcement Learning
• Hindsight Expectation Maximization for Goal-conditioned Reinforcement Learning
• Language-Conditioned Goal Generation: a New Approach to Language Grounding for RL
• An A* Curriculum Approach to Reinforcement Learning for RGBD Indoor Robot Navigation
• Weakly-Supervised Reinforcement Learning for Controllable Behavior
• GRIMGEP: Learning Progress for Robust Goal Sampling in Visual Deep Reinforcement Learning
• Generalized Decision Transformer for Offline Hindsight Information Matching
• MatchGAN: A Self-Supervised Semi-Supervised Conditional Generative Adversarial Network
• Active Inference Tree Search in Large POMDPs
• Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning
• Imagine That! Leveraging Emergent Affordances for 3D Tool Synthesis
• Active Perception and Representation for Robotic Manipulation
• Estimating Q(s,s') with Deep Deterministic Dynamics Gradients
• Skill Transfer in Deep Reinforcement Learning under Morphological Heterogeneity
• Learning to Manipulate Object Collections Using Grounded State Representations
• Disentangling causal effects for hierarchical reinforcement learning
• Outcome-Driven Reinforcement Learning via Variational Inference
• Language Conditioned Imitation Learning over Unstructured Data
• Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discovery
• Exploration via Hindsight Goal Generation
• Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
• Imaginary Hindsight Experience Replay: Curious Model-based Learning for Sparse Reward Tasks
• ABC-LMPC: Safe Sample-Based Learning MPC for Stochastic Nonlinear Dynamical Systems with Adjustable Boundary Conditions
• Visual Task Progress Estimation with Appearance Invariant Embeddings for Robot Control and Planning
• Follow the Object: Curriculum Learning for Manipulation Tasks with Imagined Goals
• An Extensible Interactive Interface for Agent Design
• Self-supervised Reinforcement Learning with Independently Controllable Subgoals
• Stein Variational Goal Generation for adaptive Exploration in Multi-Goal Reinforcement Learning
• Visual Radial Basis Q-Network
• Planning under Uncertainty to Goal Distributions
• Scaling All-Goals Updates in Reinforcement Learning Using Convolutional Neural Networks
• Towards Robust Bisimulation Metric Learning
• Sufficiently Accurate Model Learning
• Learning in Sparse Rewards settings through Quality-Diversity algorithms
• Kanerva++: extending The Kanerva Machine with differentiable, locally block allocated latent memory
• Leveraging Sequentiality in Reinforcement Learning from a Single Demonstration
• Rearranging the Environment to Maximize Energy with a Robotic Circuit Drawing
• Towards Better Understanding of Disentangled Representations via Mutual Information
• Hierarchical Policies for Cluttered-Scene Grasping with Latent Plans
• Adaptive Multi-Goal Exploration
• Learning of feature points without additional supervision improves reinforcement learning from images