Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control

Abstract

Deep reinforcement learning (RL) algorithms can learn complex robotic skills from raw sensory inputs, but have yet to achieve the kind of broad generalization and applicability demonstrated by deep learning methods in supervised domains. We present a deep RL method that is practical for real-world robotics tasks, such as robotic manipulation, and generalizes effectively to never-before-seen tasks and objects. In these settings, ground truth reward signals are typically unavailable, and we therefore propose a self-supervised model-based approach, where a predictive model learns to directly predict the future from raw sensory readings, such as camera images. At test time, we explore three distinct goal specification methods: designated pixels, where a user specifies desired object manipulation tasks by selecting particular pixels in an image and corresponding goal positions, goal images, where the desired goal state is specified with an image, and image classifiers, which define spaces of goal states. Our deep predictive models are trained using data collected autonomously and continuously by a robot interacting with hundreds of objects, without human supervision. We demonstrate that visual MPC can generalize to never-before-seen objects---both rigid and deformable---and solve a range of user-defined object manipulation tasks using the same model.

References in corpus (19)

• Adam: A Method for Stochastic Optimization
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Playing Atari with Deep Reinforcement Learning
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• Generating Videos with Scene Dynamics
• Unsupervised Learning for Physical Interaction through Video Prediction
• Uncertainty-Aware Reinforcement Learning for Collision Avoidance
• Stochastic Adversarial Video Prediction
• The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?
• SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning
• Learning to Poke by Poking: Experiential Learning of Intuitive Physics
• Recurrent Environment Simulators
• Self-Supervised Visual Planning with Temporal Skip Connections
• Video Frame Interpolation via Adaptive Separable Convolution
• Parallel Multiscale Autoregressive Density Estimation
• Few-Shot Goal Inference for Visuomotor Learning and Planning
• Learning to Fly by Crashing

Cited by in corpus (97)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
• Model-Based Reinforcement Learning for Atari
• A Review on Deep Learning Techniques for Video Prediction
• MOPO: Model-based Offline Policy Optimization
• When to Trust Your Model: Model-Based Policy Optimization
• Learning Predictive Representations for Deformable Objects Using Contrastive Estimation
• Measuring and modeling the motor system with machine learning
• SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation
• Self-Supervised Policy Adaptation during Deployment
• Alleviating Matthew Effect of Offline Reinforcement Learning in Interactive Recommendation
• Vision-driven Compliant Manipulation for Reliable, High-Precision Assembly Tasks
• Planning to Explore via Self-Supervised World Models
• COMBO: Conservative Offline Model-Based Policy Optimization
• Visual Foresight Trees for Object Retrieval from Clutter with Nonprehensile Rearrangement
• COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning
• Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation
• MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale
• Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learning
• Entity Abstraction in Visual Model-Based Reinforcement Learning
• FitVid: Overfitting in Pixel-Level Video Prediction
• Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors
• 3D Neural Scene Representations for Visuomotor Control
• Action and Perception as Divergence Minimization
• Learning Latent Plans from Play
• Learning 3D Dynamic Scene Representations for Robot Manipulation
• Offline Reinforcement Learning with Reverse Model-based Imagination
• VisuoSpatial Foresight for Multi-Step, Multi-Task Fabric Manipulation
• Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones
• High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networks
• Model-Based Offline Planning
• Model-Based Visual Planning with Self-Supervised Functional Distances
• Offline Reinforcement Learning from Images with Latent Space Models
• Causally Correct Partial Models for Reinforcement Learning
• RoboNet: Large-Scale Multi-Robot Learning
• Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generation
• Learning Dense Visual Correspondences in Simulation to Smooth and Fold Real Fabrics
• On the role of planning in model-based deep reinforcement learning
• Towards Latent Space Based Manipulation of Elastic Rods using Autoencoder Models and Robust Centerline Extractions
• Learning to Optimize in Model Predictive Control
• Learning Visible Connectivity Dynamics for Cloth Smoothing
• Experience-Embedded Visual Foresight
• Hallucinative Topological Memory for Zero-Shot Visual Planning
• 3D-OES: Viewpoint-Invariant Object-Factorized Environment Simulators
• Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation
• Learning Models as Functionals of Signed-Distance Fields for Manipulation Planning
• Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble
• Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks
• Self-Supervised Learning of State Estimation for Manipulating Deformable Linear Objects
• Grasping in the Wild:Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrations
• Learning to Predict Without Looking Ahead: World Models Without Forward Prediction
• C-Learning: Learning to Achieve Goals via Recursive Classification
• Deep Imitation Learning of Sequential Fabric Smoothing From an Algorithmic Supervisor
• Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks
• Model-Based Inverse Reinforcement Learning from Visual Demonstrations
• Control-Aware Representations for Model-based Reinforcement Learning
• Prediction, Consistency, Curvature: Representation Learning for Locally-Linear Control
• Improving Model-Based Reinforcement Learning with Internal State Representations through Self-Supervision
• VisuoSpatial Foresight for Physical Sequential Fabric Manipulation
• A Long Horizon Planning Framework for Manipulating Rigid Pointcloud Objects
• Greedy Hierarchical Variational Autoencoders for Large-Scale Video Prediction
• Occlusion resistant learning of intuitive physics from videos
• Planning Robot Motion using Deep Visual Prediction
• Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Models
• A Workflow for Offline Model-Free Robotic Reinforcement Learning
• Deep Affordance Foresight: Planning Through What Can Be Done in the Future
• Learning World Transition Model for Socially Aware Robot Navigation
• Reasoning and Generalization in RL: A Tool Use Perspective
• Model-based Reinforcement Learning for Decentralized Multiagent Rendezvous
• Motion2Vec: Semi-Supervised Representation Learning from Surgical Videos
• Active Perception and Representation for Robotic Manipulation
• Unsupervised Visual Attention and Invariance for Reinforcement Learning
• Learning Long-term Visual Dynamics with Region Proposal Interaction Networks
• Learning Multi-agent Skills for Tabular Reinforcement Learning using Factor Graphs
• Unsupervised Learning of Lagrangian Dynamics from Images for Prediction and Control
• Models, Pixels, and Rewards: Evaluating Design Trade-offs in Visual Model-Based Reinforcement Learning
• Block Contextual MDPs for Continual Learning
• Learning a Generative Transition Model for Uncertainty-Aware Robotic Manipulation
• Probably Approximately Correct Vision-Based Planning using Motion Primitives
• Language Conditioned Imitation Learning over Unstructured Data
• Learning for Deformable Linear Object Insertion Leveraging Flexibility Estimation from Visual Cues
• Structured agents for physical construction
• Explaining Conditions for Reinforcement Learning Behaviors from Real and Imagined Data
• CLOUD: Contrastive Learning of Unsupervised Dynamics
• The Information Geometry of Unsupervised Reinforcement Learning
• Simulated Mental Imagery for Robotic Task Planning
• Disentangling Controllable Object through Video Prediction Improves Visual Reinforcement Learning
• Disentangling Dense Multi-Cable Knots
• LS3: Latent Space Safe Sets for Long-Horizon Visuomotor Control of Sparse Reward Iterative Tasks
• Learning to Cope with Adversarial Attacks
• Swoosh! Rattle! Thump! -- Actions that Sound
• Cloth Manipulation Planning on Basis of Mesh Representations with Incomplete Domain Knowledge and Voxel-to-Mesh Estimation
• Learning Topological Motion Primitives for Knot Planning
• Visual Perspective Taking for Opponent Behavior Modeling
• VP-GO: a "light" action-conditioned visual prediction model
• OMG-RL:Offline Model-based Guided Reward Learning for Heparin Treatment
• Deep Variational Luenberger-type Observer for Stochastic Video Prediction