Adaptive Temporal-Difference Learning for Policy Evaluation with Per-State Uncertainty Estimates
arXiv:1906.07987
Abstract
We consider the core reinforcement-learning problem of on-policy value function approximation from a batch of trajectory data, and focus on various issues of Temporal Difference (TD) learning and Monte Carlo (MC) policy evaluation. The two methods are known to achieve complementary bias-variance trade-off properties, with TD tending to achieve lower variance but potentially higher bias. In this paper, we argue that the larger bias of TD can be a result of the amplification of local approximation errors. We address this by proposing an algorithm that adaptively switches between TD and MC in each state, thus mitigating the propagation of errors. Our method is based on learned confidence intervals that detect biases of TD estimates. We demonstrate in a variety of policy evaluation tasks that this simple adaptive algorithm performs competitively with the best approach in hindsight, suggesting that learned confidence intervals are a powerful technique for adapting policy evaluation to use TD or MC returns in a data-driven way.
References in corpus (5)
- Weight Uncertainty in Neural Networks
- Scalable Bayesian Optimization Using Deep Neural Networks
- Should one compute the Temporal Difference fix point or minimize the Bellman Residual? The unified oblique projection view
- Adaptive Lambda Least-Squares Temporal Difference Learning
- Temporal Difference Learning with Neural Networks - Study of the Leakage Propagation Problem