Learning to Combat Compounding-Error in Model-Based Reinforcement Learning

Abstract

Despite its potential to improve sample complexity versus model-free approaches, model-based reinforcement learning can fail catastrophically if the model is inaccurate. An algorithm should ideally be able to trust an imperfect model over a reasonably long planning horizon, and only rely on model-free updates when the model errors get infeasibly large. In this paper, we investigate techniques for choosing the planning horizon on a state-dependent basis, where a state's planning horizon is determined by the maximum cumulative model error around that state. We demonstrate that these state-dependent model errors can be learned with Temporal Difference methods, based on a novel approach of temporally decomposing the cumulative model errors. Experimental results show that the proposed method can successfully adapt the planning horizon to account for state-dependent model accuracy, significantly improving the efficiency of policy learning compared to model-based and model-free baselines.

References in corpus (6)

• Continuous control with deep reinforcement learning
• DeepStack: Expert-Level Artificial Intelligence in No-Limit Poker
• Benchmarking Model-Based Reinforcement Learning
• Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
• Model-based RL in Contextual Decision Processes: PAC bounds and Exponential Improvements over Model-free Approaches
• Self-Correcting Models for Model-Based Reinforcement Learning

Cited by in corpus (8)

• Deep Learning Based Simulators for the Phosphorus Removal Process Control in Wastewater Treatment via Deep Reinforcement Learning Algorithms
• Model-based Policy Optimization with Unsupervised Model Adaptation
• Selective Dyna-style Planning Under Limited Model Capacity
• Explaining Off-Policy Actor-Critic From A Bias-Variance Perspective
• Monte Carlo Tree Search in the Presence of Transition Uncertainty
• Planning with Exploration: Addressing Dynamics Bottleneck in Model-based Reinforcement Learning
• A Contraction Approach to Model-based Reinforcement Learning
• Domain Knowledge Integration By Gradient Matching For Sample-Efficient Reinforcement Learning