Calibrated Model-Based Deep Reinforcement Learning

Abstract

Estimates of predictive uncertainty are important for accurate model-based planning and reinforcement learning. However, predictive uncertainties---especially ones derived from modern deep learning systems---can be inaccurate and impose a bottleneck on performance. This paper explores which uncertainties are needed for model-based reinforcement learning and argues that good uncertainties must be calibrated, i.e. their probabilities should match empirical frequencies of predicted events. We describe a simple way to augment any model-based reinforcement learning agent with a calibrated model and show that doing so consistently improves planning, sample complexity, and exploration. On the \textsc{HalfCheetah} MuJoCo task, our system achieves state-of-the-art performance using 50\% fewer samples than the current leading approach. Our findings suggest that calibration can improve the performance of model-based reinforcement learning with minimal computational and implementation overhead.

Cited by in corpus (15)

• Verified Uncertainty Calibration
• Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantification
• PAC Confidence Sets for Deep Neural Networks via Calibrated Prediction
• Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification
• Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planning
• PAC Prediction Sets Under Covariate Shift
• Combining Pessimism with Optimism for Robust and Efficient Model-Based Deep Reinforcement Learning
• Explaining Conditions for Reinforcement Learning Behaviors from Real and Imagined Data
• Uncertainty Estimation and Calibration with Finite-State Probabilistic RNNs
• Safety Enhancement for Deep Reinforcement Learning in Autonomous Separation Assurance
• Model-based micro-data reinforcement learning: what are the crucial model properties and which model to choose?
• Deep Reinforcement Learning with Weighted Q-Learning
• On the Expressivity of Neural Networks for Deep Reinforcement Learning
• MBDP: A Model-based Approach to Achieve both Robustness and Sample Efficiency via Double Dropout Planning
• DROMO: Distributionally Robust Offline Model-based Policy Optimization