Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills

Abstract

We consider the problem of learning useful robotic skills from previously collected offline data without access to manually specified rewards or additional online exploration, a setting that is becoming increasingly important for scaling robot learning by reusing past robotic data. In particular, we propose the objective of learning a functional understanding of the environment by learning to reach any goal state in a given dataset. We employ goal-conditioned Q-learning with hindsight relabeling and develop several techniques that enable training in a particularly challenging offline setting. We find that our method can operate on high-dimensional camera images and learn a variety of skills on real robots that generalize to previously unseen scenes and objects. We also show that our method can learn to reach long-horizon goals across multiple episodes through goal chaining, and learn rich representations that can help with downstream tasks through pre-training or auxiliary objectives. The videos of our experiments can be found at https://actionable-models.github.io

References in corpus (27)

• Conservative Q-Learning for Offline Reinforcement Learning
• Model-Based Reinforcement Learning for Atari
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• Learning Latent Dynamics for Planning from Pixels
• Hindsight Experience Replay
• Stochastic Video Generation with a Learned Prior
• Visual Reinforcement Learning with Imagined Goals
• Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images
• Sim2Real Predictivity: Does Evaluation in Simulation Predict Real-World Performance?
• Successor Features for Transfer in Reinforcement Learning
• Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
• Temporal Difference Models: Model-Free Deep RL for Model-Based Control
• Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion
• Self-Imitation Learning
• Exploring Model-based Planning with Policy Networks
• Model Predictive Path Integral Control using Covariance Variable Importance Sampling
• COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning
• MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale
• Goal-conditioned Imitation Learning
• Learning to Reach Goals via Iterated Supervised Learning
• Successor Uncertainties: Exploration and Uncertainty in Temporal Difference Learning
• Broadly-Exploring, Local-Policy Trees for Long-Horizon Task Planning
• Offline Learning from Demonstrations and Unlabeled Experience
• Positive-Unlabeled Reward Learning
• Reinforcement Learning without Ground-Truth State
• Universal Value Density Estimation for Imitation Learning and Goal-Conditioned Reinforcement Learning
• Distributed Reinforcement Learning of Targeted Grasping with Active Vision for Mobile Manipulators

Cited by in corpus (9)

• ViKiNG: Vision-Based Kilometer-Scale Navigation with Geographic Hints
• Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation
• A Workflow for Offline Model-Free Robotic Reinforcement Learning
• Generalized Decision Transformer for Offline Hindsight Information Matching
• Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
• The Information Geometry of Unsupervised Reinforcement Learning
• Wish you were here: Hindsight Goal Selection for long-horizon dexterous manipulation
• Single RGB-D Camera Teleoperation for General Robotic Manipulation
• Understanding the World Through Action