Model-Based Visual Planning with Self-Supervised Functional Distances
arXiv:2012.15373
Abstract
A generalist robot must be able to complete a variety of tasks in its environment. One appealing way to specify each task is in terms of a goal observation. However, learning goal-reaching policies with reinforcement learning remains a challenging problem, particularly when hand-engineered reward functions are not available. Learned dynamics models are a promising approach for learning about the environment without rewards or task-directed data, but planning to reach goals with such a model requires a notion of functional similarity between observations and goal states. We present a self-supervised method for model-based visual goal reaching, which uses both a visual dynamics model as well as a dynamical distance function learned using model-free reinforcement learning. Our approach learns entirely using offline, unlabeled data, making it practical to scale to large and diverse datasets. In our experiments, we find that our method can successfully learn models that perform a variety of tasks at test-time, moving objects amid distractors with a simulated robotic arm and even learning to open and close a drawer using a real-world robot. In comparisons, we find that this approach substantially outperforms both model-free and model-based prior methods. Videos and visualizations are available here: http://sites.google.com/berkeley.edu/mbold.
References in corpus (18)
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
- Conservative Q-Learning for Offline Reinforcement Learning
- Learning Latent Dynamics for Planning from Pixels
- Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
- Behavior Regularized Offline Reinforcement Learning
- Stochastic Adversarial Video Prediction
- Dream to Control: Learning Behaviors by Latent Imagination
- Exploring Model-based Planning with Policy Networks
- Stochastic Variational Video Prediction
- AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
- Planning with Goal-Conditioned Policies
- Robustness via Retrying: Closed-Loop Robotic Manipulation with Self-Supervised Learning
- Self-supervised Learning of Image Embedding for Continuous Control
- Learning to Reach Goals via Iterated Supervised Learning
- Model-Based Offline Planning
- Hallucinative Topological Memory for Zero-Shot Visual Planning
- Experience-Embedded Visual Foresight