Correcting Momentum in Temporal Difference Learning
arXiv:2106.03955
Abstract
A common optimization tool used in deep reinforcement learning is momentum, which consists in accumulating and discounting past gradients, reapplying them at each iteration. We argue that, unlike in supervised learning, momentum in Temporal Difference (TD) learning accumulates gradients that become doubly stale: not only does the gradient of the loss change due to parameter updates, the loss itself changes due to bootstrapping. We first show that this phenomenon exists, and then propose a first-order correction term to momentum. We show that this correction term improves sample efficiency in policy evaluation by correcting target value drift. An important insight of this work is that deep RL methods are not always best served by directly importing techniques from the supervised setting.
arXiv commentNeurIPS Deep RL Workshop 2020
References in corpus (13)
- Adam: A Method for Stochastic Optimization
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Playing Atari with Deep Reinforcement Learning
- Pruning Convolutional Neural Networks for Resource Efficient Inference
- Rainbow: Combining Improvements in Deep Reinforcement Learning
- Assessing Generalization in Deep Reinforcement Learning
- What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study
- Generalization and Regularization in DQN
- Towards Characterizing Divergence in Deep Q-Learning
- Accelerated Gradient Temporal Difference Learning
- Adaptive Temporal Difference Learning with Linear Function Approximation
- Deep Residual Reinforcement Learning
- TDprop: Does Jacobi Preconditioning Help Temporal Difference Learning?