A Contraction Approach to Model-based Reinforcement Learning

Abstract

Despite its experimental success, Model-based Reinforcement Learning still lacks a complete theoretical understanding. To this end, we analyze the error in the cumulative reward using a contraction approach. We consider both stochastic and deterministic state transitions for continuous (non-discrete) state and action spaces. This approach doesn't require strong assumptions and can recover the typical quadratic error to the horizon. We prove that branched rollouts can reduce this error and are essential for deterministic transitions to have a Bellman contraction. Our analysis of policy mismatch error also applies to Imitation Learning. In this case, we show that GAN-type learning has an advantage over Behavioral Cloning when its discriminator is well-trained.

arXiv commentThe 24th International Conference on Artificial Intelligence and Statistics (AISTATS) 2021

References in corpus (12)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Spectral Normalization for Generative Adversarial Networks
• Continuous Deep Q-Learning with Model-based Acceleration
• Model-Ensemble Trust-Region Policy Optimization
• MOPO: Model-based Offline Policy Optimization
• Benchmarking Model-Based Reinforcement Learning
• Provably Efficient Reinforcement Learning with Linear Function Approximation
• Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• Model-Based Reinforcement Learning via Meta-Policy Optimization
• Clipped Action Policy Gradient
• Learning to Combat Compounding-Error in Model-Based Reinforcement Learning