Combating the Compounding-Error Problem with a Multi-step Model

Abstract

Model-based reinforcement learning is an appealing framework for creating agents that learn, plan, and act in sequential environments. Model-based algorithms typically involve learning a transition model that takes a state and an action and outputs the next state---a one-step model. This model can be composed with itself to enable predicting multiple steps into the future, but one-step prediction errors can get magnified, leading to unacceptable inaccuracy. This compounding-error problem plagues planning and undermines model-based reinforcement learning. In this paper, we address the compounding-error problem by introducing a multi-step model that directly outputs the outcome of executing a sequence of actions. Novel theoretical and empirical results indicate that the multi-step model is more conducive to efficient value-function estimation, and it yields better action selection compared to the one-step model. These results make a strong case for using multi-step models in the context of model-based reinforcement learning.

References in corpus (12)

• Playing Atari with Deep Reinforcement Learning
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Safe Model-based Reinforcement Learning with Stability Guarantees
• The Predictron: End-To-End Learning and Planning
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• Dyna-Style Planning with Linear Function Approximation and Prioritized Sweeping
• Model-based RL in Contextual Decision Processes: PAC bounds and Exponential Improvements over Model-free Approaches
• On the Sample Complexity of Reinforcement Learning with a Generative Model
• Compositional Planning Using Optimal Option Models
• Learning Multimodal Transition Dynamics for Model-Based Reinforcement Learning
• Self-Correcting Models for Model-Based Reinforcement Learning
• Equivalence Between Wasserstein and Value-Aware Loss for Model-based Reinforcement Learning

Cited by in corpus (9)

• Deep Learning Based Simulators for the Phosphorus Removal Process Control in Wastewater Treatment via Deep Reinforcement Learning Algorithms
• Robust Model-based Reinforcement Learning for Autonomous Greenhouse Control
• Causally Correct Partial Models for Reinforcement Learning
• Model-based Policy Optimization with Unsupervised Model Adaptation
• Model Imitation for Model-Based Reinforcement Learning
• Sample Efficient Reinforcement Learning via Model-Ensemble Exploration and Exploitation
• MBDP: A Model-based Approach to Achieve both Robustness and Sample Efficiency via Double Dropout Planning
• Generative Temporal Difference Learning for Infinite-Horizon Prediction
• Planning with Exploration: Addressing Dynamics Bottleneck in Model-based Reinforcement Learning