Online and Offline Reinforcement Learning by Planning with a Learned Model

Abstract

Learning efficiently from small amounts of data has long been the focus of model-based reinforcement learning, both for the online case when interacting with the environment and the offline case when learning from a fixed dataset. However, to date no single unified algorithm could demonstrate state-of-the-art results in both settings. In this work, we describe the Reanalyse algorithm which uses model-based policy and value improvement operators to compute new improved training targets on existing data points, allowing efficient learning for data budgets varying by several orders of magnitude. We further show that Reanalyse can also be used to learn entirely from demonstrations without any environment interactions, as in the case of offline Reinforcement Learning (offline RL). Combining Reanalyse with the MuZero algorithm, we introduce MuZero Unplugged, a single unified algorithm for any data budget, including offline RL. In contrast to previous work, our algorithm does not require any special adaptations for the off-policy or offline RL settings. MuZero Unplugged sets new state-of-the-art results in the RL Unplugged offline RL benchmark as well as in the online RL benchmark of Atari in the standard 200 million frame setting.

References in corpus (14)

• Improving neural networks by preventing co-adaptation of feature detectors
• Scaling Laws for Neural Language Models
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Solving Rubik's Cube with a Robot Hand
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• DeepMind Control Suite
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Behavior Regularized Offline Reinforcement Learning
• Distributional Reinforcement Learning with Quantile Regression
• Critic Regularized Regression
• Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
• Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning
• Offline Reinforcement Learning from Images with Latent Space Models
• Learning and Planning in Complex Action Spaces

Cited by in corpus (10)

• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• Muesli: Combining Improvements in Policy Optimization
• The Benchmark Lottery
• Scalable Online Planning via Reinforcement Learning Fine-Tuning
• Procedural Generalization by Planning with Self-Supervised World Models
• Sample Efficient Reinforcement Learning with Partial Dynamics Knowledge
• Improve Agents without Retraining: Parallel Tree Search with Off-Policy Correction
• The Difficulty of Passive Learning in Deep Reinforcement Learning
• Self-Consistent Models and Values
• Proper Value Equivalence