COMBO: Conservative Offline Model-Based Policy Optimization

Abstract

Model-based algorithms, which learn a dynamics model from logged experience and perform some sort of pessimistic planning under the learned model, have emerged as a promising paradigm for offline reinforcement learning (offline RL). However, practical variants of such model-based algorithms rely on explicit uncertainty quantification for incorporating pessimism. Uncertainty estimation with complex models, such as deep neural networks, can be difficult and unreliable. We overcome this limitation by developing a new model-based offline RL algorithm, COMBO, that regularizes the value function on out-of-support state-action tuples generated via rollouts under the learned model. This results in a conservative estimate of the value function for out-of-support state-action tuples, without requiring explicit uncertainty estimation. We theoretically show that our method optimizes a lower bound on the true policy value, that this bound is tighter than that of prior methods, and our approach satisfies a policy improvement guarantee in the offline setting. Through experiments, we find that COMBO consistently performs as well or better as compared to prior offline model-free and model-based methods on widely studied offline RL benchmarks, including image-based tasks.

NeurIPS 2021

References in corpus (14)

• Conservative Q-Learning for Offline Reinforcement Learning
• DeepMind Control Suite
• Behavior Regularized Offline Reinforcement Learning
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• AlgaeDICE: Policy Gradient from Arbitrary Experience
• Safe Policy Improvement by Minimizing Robust Baseline Regret
• Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
• Information-Theoretic Considerations in Batch Reinforcement Learning
• Off-Policy Policy Gradient with State Distribution Correction
• COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning
• Provably Good Batch Reinforcement Learning Without Great Exploration
• Model-Based Offline Planning
• Offline Reinforcement Learning from Images with Latent Space Models

Cited by in corpus (16)

• What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
• Alleviating Matthew Effect of Offline Reinforcement Learning in Interactive Recommendation
• Offline Reinforcement Learning with Reverse Model-based Imagination
• Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
• ORL-AUDITOR: Dataset Auditing in Offline Deep Reinforcement Learning
• Corruption-Robust Offline Reinforcement Learning
• Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage
• Instabilities of Offline RL with Pre-Trained Neural Representation
• OER: Offline Experience Replay for Continual Offline Reinforcement Learning
• Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
• Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulation
• Provably Efficient Generative Adversarial Imitation Learning for Online and Offline Setting with Linear Function Approximation
• UMBRELLA: Uncertainty-Aware Model-Based Offline Reinforcement Learning Leveraging Planning
• Learning MDPs from Features: Predict-Then-Optimize for Sequential Decision Problems by Reinforcement Learning
• DCUR: Data Curriculum for Teaching via Samples with Reinforcement Learning
• Offline Inverse Reinforcement Learning