Model-Based Offline Planning

Abstract

Offline learning is a key part of making reinforcement learning (RL) useable in real systems. Offline RL looks at scenarios where there is data from a system's operation, but no direct access to the system when learning a policy. Recent work on training RL policies from offline data has shown results both with model-free policies learned directly from the data, or with planning on top of learnt models of the data. Model-free policies tend to be more performant, but are more opaque, harder to command externally, and less easy to integrate into larger systems. We propose an offline learner that generates a model that can be used to control the system directly through planning. This allows us to have easily controllable policies directly from data, without ever interacting with the system. We show the performance of our algorithm, Model-Based Offline Planning (MBOP) on a series of robotics-inspired tasks, and demonstrate its ability leverage planning to respect environmental constraints. We are able to find near-optimal polices for certain simulated systems from as little as 50 seconds of real-time system interaction, and create zero-shot goal-conditioned policies on a series of environments. An accompanying video can be found here: https://youtu.be/nxGGHdZOFts

References in corpus (21)

• Continuous control with deep reinforcement learning
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• DeepMind Control Suite
• Learning Latent Dynamics for Planning from Pixels
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
• Behavior Regularized Offline Reinforcement Learning
• MOPO: Model-based Offline Policy Optimization
• Maximum a Posteriori Policy Optimisation
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• MOReL : Model-Based Offline Reinforcement Learning
• Dream to Control: Learning Behaviors by Latent Imagination
• DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
• Critic Regularized Regression
• Exploring Model-based Planning with Policy Networks
• Model Predictive Path Integral Control using Covariance Variable Importance Sampling
• Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control
• A Game Theoretic Framework for Model Based Reinforcement Learning
• An empirical investigation of the challenges of real-world reinforcement learning
• Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
• Adaptive Online Planning for Continual Lifelong Learning

Cited by in corpus (13)

• COMBO: Conservative Offline Model-Based Policy Optimization
• An empirical investigation of the challenges of real-world reinforcement learning
• Offline Reinforcement Learning with Fisher Divergence Critic Regularization
• Offline Reinforcement Learning with Reverse Model-based Imagination
• Model-Based Visual Planning with Self-Supervised Functional Distances
• Offline Reinforcement Learning from Images with Latent Space Models
• Reset-Free Lifelong Learning with Skill-Space Planning
• Offline Reinforcement Learning with Value-based Episodic Memory
• Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
• Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
• BRAC+: Improved Behavior Regularized Actor Critic for Offline Reinforcement Learning
• Towards Deeper Deep Reinforcement Learning with Spectral Normalization
• Learning Off-Policy with Online Planning