When to use parametric models in reinforcement learning?

Abstract

We examine the question of when and how parametric models are most useful in reinforcement learning. In particular, we look at commonalities and differences between parametric models and experience replay. Replay-based learning algorithms share important traits with model-based approaches, including the ability to plan: to use more computation without additional data to improve predictions and behaviour. We discuss when to expect benefits from either approach, and interpret prior work in this context. We hypothesise that, under suitable conditions, replay-based algorithms should be competitive to or better than model-based algorithms if the model is used only to generate fictional transitions from observed states for an update rule that is otherwise model-free. We validated this hypothesis on Atari 2600 video games. The replay-based algorithm attained state-of-the-art data efficiency, improving over prior results with parametric models.

Cited by in corpus (33)

• Model-Based Reinforcement Learning for Atari
• CURL: Contrastive Unsupervised Representations for Reinforcement Learning
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• An Optimistic Perspective on Offline Reinforcement Learning
• Data-Efficient Reinforcement Learning with Self-Predictive Representations
• SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning
• Deep Reinforcement Learning at the Edge of the Statistical Precipice
• Pretraining Representations for Data-Efficient Reinforcement Learning
• Randomized Ensembled Double Q-Learning: Learning Fast Without a Model
• Offline Reinforcement Learning with Reverse Model-based Imagination
• Return-Based Contrastive Representation Learning for Reinforcement Learning
• On the model-based stochastic value gradient for continuous reinforcement learning
• On the role of planning in model-based deep reinforcement learning
• Muesli: Combining Improvements in Policy Optimization
• How to Learn a Useful Critic? Model-based Action-Gradient-Estimator Policy Optimization
• Behavior From the Void: Unsupervised Active Pre-Training
• Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planning
• PlayVirtual: Augmenting Cycle-Consistent Virtual Trajectories for Reinforcement Learning
• Local Search for Policy Iteration in Continuous Control
• Learning Markov State Abstractions for Deep Reinforcement Learning
• Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddings
• Selective Dyna-style Planning Under Limited Model Capacity
• Learning Retrospective Knowledge with Reverse Reinforcement Learning
• Biologically inspired architectures for sample-efficient deep reinforcement learning
• Learning to Sample with Local and Global Contexts in Experience Replay Buffer
• Benchmarking the Spectrum of Agent Capabilities
• Improving Experience Replay through Modeling of Similar Transitions' Sets
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• Self-Consistent Models and Values
• Improved Memories Learning
• Model-free Policy Learning with Reward Gradients
• Planning with Expectation Models for Control
• The Efficiency Misnomer