On-Policy Model Errors in Reinforcement Learning

Abstract

Model-free reinforcement learning algorithms can compute policy gradients given sampled environment transitions, but require large amounts of data. In contrast, model-based methods can use the learned model to generate new data, but model errors and bias can render learning unstable or suboptimal. In this paper, we present a novel method that combines real-world data and a learned model in order to get the best of both worlds. The core idea is to exploit the real-world data for on-policy predictions and use the learned model only to generalize to different actions. Specifically, we use the data as time-dependent on-policy correction terms on top of a learned model, to retain the ability to generate data without accumulating errors over long prediction horizons. We motivate this method theoretically and show that it counteracts an error term for model-based policy improvement. Experiments on MuJoCo- and PyBullet-benchmarks show that our method can drastically improve existing model-based approaches without introducing additional tuning parameters.

Published at The Tenth International Conference on Learning Representations (ICLR 2022)

References in corpus (14)

• Neural Architecture Search with Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Statistical Aspects of Wasserstein Distances
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Model-Based Reinforcement Learning for Atari
• Model-Ensemble Trust-Region Policy Optimization
• MOPO: Model-based Offline Policy Optimization
• Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• When to Trust Your Model: Model-Based Policy Optimization
• Combining Model-Based and Model-Free Updates for Trajectory-Centric Reinforcement Learning
• Model-Augmented Actor-Critic: Backpropagating through Paths
• On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning
• Predictor-Corrector Policy Optimization