TDprop: Does Jacobi Preconditioning Help Temporal Difference Learning?

Abstract

We investigate whether Jacobi preconditioning, accounting for the bootstrap term in temporal difference (TD) learning, can help boost performance of adaptive optimizers. Our method, TDprop, computes a per parameter learning rate based on the diagonal preconditioning of the TD update rule. We show how this can be used in both $n$-step returns and TD($\lambda$). Our theoretical findings demonstrate that including this additional preconditioning information is, surprisingly, comparable to normal semi-gradient TD if the optimal learning rate is found for both via a hyperparameter search. In Deep RL experiments using Expected SARSA, TDprop meets or exceeds the performance of Adam in all tested games under near-optimal learning rates, but a well-tuned SGD can yield similar improvements -- matching our theory. Our findings suggest that Jacobi preconditioning may improve upon typical adaptive optimization methods in Deep RL, but despite incorporating additional information from the TD bootstrap term, may not always be better than SGD.

arXiv commentPresented at the Theoretical Foundations of Reinforcement Learning workshop at ICML 2020

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Playing Atari with Deep Reinforcement Learning
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• New insights and perspectives on the natural gradient method
• No More Pesky Learning Rates
• Implicit Quantile Networks for Distributional Reinforcement Learning
• Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning
• Dopamine: A Research Framework for Deep Reinforcement Learning
• Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
• Equilibrated adaptive learning rates for non-convex optimization
• Where Did My Optimum Go?: An Empirical Analysis of Gradient Descent Optimization in Policy Gradient Methods
• Zap Q-Learning With Nonlinear Function Approximation
• Learning Feature Relevance Through Step Size Adaptation in Temporal-Difference Learning
• BackPACK: Packing more into backprop

Cited by in corpus (2)

• Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspective
• Correcting Momentum in Temporal Difference Learning