Offline Reinforcement Learning from Images with Latent Space Models

Abstract

Offline reinforcement learning (RL) refers to the problem of learning policies from a static dataset of environment interactions. Offline RL enables extensive use and re-use of historical datasets, while also alleviating safety concerns associated with online exploration, thereby expanding the real-world applicability of RL. Most prior work in offline RL has focused on tasks with compact state representations. However, the ability to learn directly from rich observation spaces like images is critical for real-world applications such as robotics. In this work, we build on recent advances in model-based algorithms for offline RL, and extend them to high-dimensional visual observation spaces. Model-based offline RL algorithms have achieved state of the art results in state based tasks and have strong theoretical guarantees. However, they rely crucially on the ability to quantify uncertainty in the model predictions, which is particularly challenging with image observations. To overcome this challenge, we propose to learn a latent-state dynamics model, and represent the uncertainty in the latent space. Our approach is both tractable in practice and corresponds to maximizing a lower bound of the ELBO in the unknown POMDP. In experiments on a range of challenging image-based locomotion and manipulation tasks, we find that our algorithm significantly outperforms previous offline model-free RL methods as well as state-of-the-art online visual model-based RL methods. Moreover, we also find that our approach excels on an image-based drawer closing task on a real robot using a pre-existing dataset. All results including videos can be found online at https://sites.google.com/view/lompo/ .

References in corpus (29)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Conservative Q-Learning for Offline Reinforcement Learning
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Learning Latent Dynamics for Planning from Pixels
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Visual Reinforcement Learning with Imagined Goals
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
• MOPO: Model-based Offline Policy Optimization
• Reinforcement Learning with Augmented Data
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• MOReL : Model-Based Offline Reinforcement Learning
• Unsupervised Predictive Memory in a Goal-Directed Agent
• SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• AlgaeDICE: Policy Gradient from Arbitrary Experience
• Contrastive Learning of Structured World Models
• DeepMDP: Learning Continuous Latent Space Models for Representation Learning
• Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model
• A Game Theoretic Framework for Model Based Reinforcement Learning
• Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
• Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning
• Off-Policy Policy Gradient with State Distribution Correction
• Provably Good Batch Reinforcement Learning Without Great Exploration
• Robust Locally-Linear Controllable Embedding
• Model-Based Offline Planning
• Experience-Embedded Visual Foresight
• Scaling Robot Supervision to Hundreds of Hours with RoboTurk: Robotic Manipulation Dataset through Human Reasoning and Dexterity

Cited by in corpus (7)

• COMBO: Conservative Offline Model-Based Policy Optimization
• FitVid: Overfitting in Pixel-Level Video Prediction
• Online and Offline Reinforcement Learning by Planning with a Learned Model
• Offline Reinforcement Learning with Reverse Model-based Imagination
• Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks
• Instabilities of Offline RL with Pre-Trained Neural Representation
• Visual Adversarial Imitation Learning using Variational Models