Kalman meets Bellman: Improving Policy Evaluation through Value Tracking
arXiv:2002.07171
Abstract
Policy evaluation is a key process in Reinforcement Learning (RL). It assesses a given policy by estimating the corresponding value function. When using parameterized value functions, common approaches minimize the sum of squared Bellman temporal-difference errors and receive a point-estimate for the parameters. Kalman-based and Gaussian-processes based frameworks were suggested to evaluate the policy by treating the value as a random variable. These frameworks can learn uncertainties over the value parameters and exploit them for policy exploration. When adopting these frameworks to solve deep RL tasks, several limitations are revealed: excessive computations in each optimization step, difficulty with handling batches of samples which slows training and the effect of memory in stochastic environments which prevents off-policy learning. In this work, we discuss these limitations and propose to overcome them by an alternative general framework, based on the extended Kalman filter. We devise an optimization method, called Kalman Optimization for Value Approximation (KOVA) that can be incorporated as a policy evaluation component in policy optimization algorithms. KOVA minimizes a regularized objective function that concerns both parameter and noisy return uncertainties. We analyze the properties of KOVA and present its performance on deep RL control tasks.
arXiv admin note: substantial text overlap with arXiv:1901.07860
References in corpus (25)
- Adam: A Method for Stochastic Optimization
- Continuous control with deep reinforcement learning
- ADADELTA: An Adaptive Learning Rate Method
- Playing Atari with Deep Reinforcement Learning
- An overview of gradient descent optimization algorithms
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Trust Region Policy Optimization
- High-Dimensional Continuous Control Using Generalized Advantage Estimation
- Weight Uncertainty in Neural Networks
- Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
- New insights and perspectives on the natural gradient method
- A Distributional Perspective on Reinforcement Learning
- Deep Exploration via Randomized Value Functions
- Randomized Prior Functions for Deep Reinforcement Learning
- Kalman Temporal Differences
- Generalization and Exploration via Randomized Value Functions
- Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam
- The Uncertainty Bellman Equation and Exploration
- Bayesian policy gradient and actor-critic algorithms
- Shallow Updates for Deep Reinforcement Learning
- Successor Uncertainties: Exploration and Uncertainty in Temporal Difference Learning
- Batch Kalman Normalization: Towards Training Deep Neural Networks with Micro-Batches
- A Kernel Loss for Solving the Bellman Equation
- Kalman Gradient Descent: Adaptive Variance Reduction in Stochastic Optimization
- Directly Estimating the Variance of the λ-Return Using Temporal-Difference Methods