MOPO: Model-based Offline Policy Optimization

Abstract

Offline reinforcement learning (RL) refers to the problem of learning policies entirely from a large batch of previously collected data. This problem setting offers the promise of utilizing such datasets to acquire policies without any costly or dangerous active exploration. However, it is also challenging, due to the distributional shift between the offline training data and those states visited by the learned policy. Despite significant recent progress, the most successful prior methods are model-free and constrain the policy to the support of data, precluding generalization to unseen states. In this paper, we first observe that an existing model-based RL algorithm already produces significant gains in the offline setting compared to model-free approaches. However, standard model-based RL methods, designed for the online setting, do not provide an explicit mechanism to avoid the offline setting's distributional shift issue. Instead, we propose to modify the existing model-based RL methods by applying them with rewards artificially penalized by the uncertainty of the dynamics. We theoretically show that the algorithm maximizes a lower bound of the policy's return under the true MDP. We also characterize the trade-off between the gain and risk of leaving the support of the batch data. Our algorithm, Model-based Offline Policy Optimization (MOPO), outperforms standard model-based RL algorithms and prior state-of-the-art model-free offline RL algorithms on existing offline RL benchmarks and two challenging continuous control tasks that require generalizing from data collected for a different task. The code is available at https://github.com/tianheyu927/mopo.

arXiv commentNeurIPS 2020. First two authors contributed equally. Last two authors advised equally

References in corpus (30)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Spectral Normalization for Generative Adversarial Networks
• Trust Region Policy Optimization
• Addressing Function Approximation Error in Actor-Critic Methods
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Accurate Uncertainties for Deep Learning Using Calibrated Regression
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• Model-Ensemble Trust-Region Policy Optimization
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
• Behavior Regularized Offline Reinforcement Learning
• Benchmarking Model-Based Reinforcement Learning
• Quantifying Generalization in Reinforcement Learning
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• MOReL : Model-Based Offline Reinforcement Learning
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• Model-Based Reinforcement Learning via Meta-Policy Optimization
• Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic
• On integral probability metrics, ϕ-divergences and binary classification
• A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning
• AlgaeDICE: Policy Gradient from Arbitrary Experience
• Exploring Model-based Planning with Policy Networks
• Deep Dynamics Models for Learning Dexterous Manipulation
• Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Bounds
• Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning
• DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction
• Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margin
• The Effect of Planning Shape on Dyna-style Planning in High-dimensional State Spaces

Cited by in corpus (81)

• On the Opportunities and Risks of Foundation Models
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• MOReL : Model-Based Offline Reinforcement Learning
• QPLEX: Duplex Dueling Multi-Agent Q-Learning
• What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
• Real-World Robot Applications of Foundation Models: A Review
• Alleviating Matthew Effect of Offline Reinforcement Learning in Interactive Recommendation
• COMBO: Conservative Offline Model-Based Policy Optimization
• An empirical investigation of the challenges of real-world reinforcement learning
• Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
• Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantification
• COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning
• Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEs
• Is Pessimism Provably Efficient for Offline RL?
• The Importance of Pessimism in Fixed-Dataset Policy Optimization
• Mastering Atari with Discrete World Models
• Representation Matters: Offline Pretraining for Sequential Decision Making
• Offline Reinforcement Learning with Reverse Model-based Imagination
• Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning
• Model-Based Offline Planning
• Dropout Q-Functions for Doubly Efficient Reinforcement Learning
• CoinDICE: Off-Policy Confidence Interval Estimation
• Offline Reinforcement Learning from Images with Latent Space Models
• Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning
• Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification
• Risk-Averse Offline Reinforcement Learning
• Bellman-consistent Pessimism for Offline Reinforcement Learning
• Probabilistic design of optimal sequential decision-making algorithms in learning and control
• Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
• Reset-Free Lifelong Learning with Skill-Space Planning
• Challenges of Applying Deep Reinforcement Learning in Dynamic Dispatching
• Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization
• ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
• Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environment
• Offline Model-Based Optimization via Normalized Maximum Likelihood Estimation
• TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations
• Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage
• Corruption-Robust Offline Reinforcement Learning
• MUSBO: Model-based Uncertainty Regularized and Sample Efficient Batch Optimization for Deployment Constrained Reinforcement Learning
• Continuous Doubly Constrained Batch Reinforcement Learning
• A Workflow for Offline Model-Free Robotic Reinforcement Learning
• Average-Reward Off-Policy Policy Evaluation with Function Approximation
• Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violations
• Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage
• Instabilities of Offline RL with Pre-Trained Neural Representation
• Offline Meta-level Model-based Reinforcement Learning Approach for Cold-Start Recommendation
• Offline reinforcement learning with uncertainty for treatment strategies in sepsis
• Provable Model-based Nonlinear Bandit and Reinforcement Learning: Shelve Optimism, Embrace Virtual Curvature
• Offline Reinforcement Learning with Pseudometric Learning
• S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning
• Representation Learning for Online and Offline RL in Low-rank MDPs
• Visual Adversarial Imitation Learning using Variational Models
• Regularized Behavior Value Estimation
• Learning Dynamics Models for Model Predictive Agents
• Meta-Model-Based Meta-Policy Optimization
• OER: Offline Experience Replay for Continual Offline Reinforcement Learning
• Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
• Personalized Product Assortment with Real-time 3D Perception and Bayesian Payoff Estimation
• Centralized Model and Exploration Policy for Multi-Agent RL
• Model-Invariant State Abstractions for Model-Based Reinforcement Learning
• PerSim: Data-Efficient Offline Reinforcement Learning with Heterogeneous Agents via Personalized Simulators
• Simulation and Model Checking for Close to Realtime Overtaking Planning
• You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL
• Towards Deeper Deep Reinforcement Learning with Spectral Normalization
• A Free Lunch from the Noise: Provable and Practical Exploration for Representation Learning
• Near-Optimal Reward-Free Exploration for Linear Mixture MDPs with Plug-in Solver
• Reducing Conservativeness Oriented Offline Reinforcement Learning
• Provably Efficient Generative Adversarial Imitation Learning for Online and Offline Setting with Linear Function Approximation
• The Difficulty of Passive Learning in Deep Reinforcement Learning
• Learning Off-Policy with Online Planning
• Curriculum Offline Imitation Learning
• UMBRELLA: Uncertainty-Aware Model-Based Offline Reinforcement Learning Leveraging Planning
• Pessimistic Model Selection for Offline Deep Reinforcement Learning
• Reward Balancing Revisited: Enhancing Offline Reinforcement Learning for Recommender Systems
• On-Policy Model Errors in Reinforcement Learning
• Uniformly Conservative Exploration in Reinforcement Learning
• OMG-RL:Offline Model-based Guided Reward Learning for Heparin Treatment
• Active Offline Policy Selection
• Offline Inverse Reinforcement Learning
• A Contraction Approach to Model-based Reinforcement Learning
• Optimal Uniform OPE and Model-based Offline Reinforcement Learning in Time-Homogeneous, Reward-Free and Task-Agnostic Settings