MOReL : Model-Based Offline Reinforcement Learning

Abstract

In offline reinforcement learning (RL), the goal is to learn a highly rewarding policy based solely on a dataset of historical interactions with the environment. The ability to train RL policies offline can greatly expand the applicability of RL, its data efficiency, and its experimental velocity. Prior work in offline RL has been confined almost exclusively to model-free RL approaches. In this work, we present MOReL, an algorithmic framework for model-based offline RL. This framework consists of two steps: (a) learning a pessimistic MDP (P-MDP) using the offline dataset; and (b) learning a near-optimal policy in this P-MDP. The learned P-MDP has the property that for any policy, the performance in the real environment is approximately lower-bounded by the performance in the P-MDP. This enables it to serve as a good surrogate for purposes of policy evaluation and learning, and overcome common pitfalls of model-based RL like model exploitation. Theoretically, we show that MOReL is minimax optimal (up to log factors) for offline RL. Through experiments, we show that MOReL matches or exceeds state-of-the-art results in widely studied offline RL benchmarks. Moreover, the modular design of MOReL enables future advances in its components (e.g. generative modeling, uncertainty estimation, planning etc.) to directly translate into advances for offline RL.

arXiv commentFirst two authors contributed equally. Published at NeurIPS 2020. After publication at NeurIPS 2020, (1) D4RL benchmark results have been added; (2) hyper-parameter ablation studies have been added; (3) scope of Lemma 3 has been extended

References in corpus (33)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Addressing Function Approximation Error in Actor-Critic Methods
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Conservative Q-Learning for Offline Reinforcement Learning
• Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation Algorithms
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Off-Policy Deep Reinforcement Learning without Exploration
• Model-Ensemble Trust-Region Policy Optimization
• Behavior Regularized Offline Reinforcement Learning
• MOPO: Model-based Offline Policy Optimization
• Benchmarking Model-Based Reinforcement Learning
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• When to Trust Your Model: Model-Based Policy Optimization
• DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
• Randomized Prior Functions for Deep Reinforcement Learning
• Deep Reinforcement Learning and the Deadly Triad
• AlgaeDICE: Policy Gradient from Arbitrary Experience
• Exploring Model-based Planning with Policy Networks
• Agnostic System Identification for Model-Based Reinforcement Learning
• Safe Policy Improvement by Minimizing Robust Baseline Regret
• A Game Theoretic Framework for Model Based Reinforcement Learning
• Information-Theoretic Considerations in Batch Reinforcement Learning
• Off-Policy Policy Gradient with State Distribution Correction
• Consistent On-Line Off-Policy Evaluation
• End-to-End Offline Goal-Oriented Dialog Policy Learning via Policy Gradient
• Reinforcement Learning via Fenchel-Rockafellar Duality
• Top-K Off-Policy Correction for a REINFORCE Recommender System
• P3O: Policy-on Policy-off Policy Optimization

Cited by in corpus (48)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• MOPO: Model-based Offline Policy Optimization
• What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
• Alleviating Matthew Effect of Offline Reinforcement Learning in Interactive Recommendation
• COMBO: Conservative Offline Model-Based Policy Optimization
• An empirical investigation of the challenges of real-world reinforcement learning
• Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
• Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEs
• Is Pessimism Provably Efficient for Offline RL?
• The Importance of Pessimism in Fixed-Dataset Policy Optimization
• Model-Based Offline Planning
• Offline Reinforcement Learning from Images with Latent Space Models
• Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning
• Bellman-consistent Pessimism for Offline Reinforcement Learning
• Probabilistic design of optimal sequential decision-making algorithms in learning and control
• Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
• PLAS: Latent Action Space for Offline Reinforcement Learning
• Reset-Free Lifelong Learning with Skill-Space Planning
• Offline Reinforcement Learning with Value-based Episodic Memory
• ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
• TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations
• Batch-Constrained Distributional Reinforcement Learning for Session-based Recommendation
• ORL-AUDITOR: Dataset Auditing in Offline Deep Reinforcement Learning
• Corruption-Robust Offline Reinforcement Learning
• A Workflow for Offline Model-Free Robotic Reinforcement Learning
• Average-Reward Off-Policy Policy Evaluation with Function Approximation
• Continuous Doubly Constrained Batch Reinforcement Learning
• S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning
• Offline Meta-level Model-based Reinforcement Learning Approach for Cold-Start Recommendation
• Learning Dynamics Models for Model Predictive Agents
• Evaluating the progress of Deep Reinforcement Learning in the real world: aligning domain-agnostic and domain-specific research
• Regularized Behavior Value Estimation
• Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
• Model-Invariant State Abstractions for Model-Based Reinforcement Learning
• OER: Offline Experience Replay for Continual Offline Reinforcement Learning
• MobILE: Model-Based Imitation Learning From Observation Alone
• PerSim: Data-Efficient Offline Reinforcement Learning with Heterogeneous Agents via Personalized Simulators
• Near-Optimal Reward-Free Exploration for Linear Mixture MDPs with Plug-in Solver
• Learning MDPs from Features: Predict-Then-Optimize for Sequential Decision Problems by Reinforcement Learning
• Offline RL With Resource Constrained Online Deployment
• Learning Off-Policy with Online Planning
• Reducing Conservativeness Oriented Offline Reinforcement Learning
• Active Offline Policy Selection
• Optimal Uniform OPE and Model-based Offline Reinforcement Learning in Time-Homogeneous, Reward-Free and Task-Agnostic Settings
• DROMO: Distributionally Robust Offline Model-based Policy Optimization
• Offline Inverse Reinforcement Learning
• Video2Skill: Adapting Events in Demonstration Videos to Skills in an Environment using Cyclic MDP Homomorphisms
• Variance-Aware Off-Policy Evaluation with Linear Function Approximation