Temporal-difference learning with nonlinear function approximation: lazy training and mean field regimes
arXiv:1905.10917
Abstract
We discuss the approximation of the value function for infinite-horizon discounted Markov Reward Processes (MRP) with nonlinear functions trained with the Temporal-Difference (TD) learning algorithm. We first consider this problem under a certain scaling of the approximating function, leading to a regime called lazy training. In this regime, the parameters of the model vary only slightly during the learning process, a feature that has recently been observed in the training of neural networks, where the scaling we study arises naturally, implicit in the initialization of their parameters. Both in the under- and over-parametrized frameworks, we prove exponential convergence to local, respectively global minimizers of the above algorithm in the lazy training regime. We then compare this scaling of the parameters to the mean-field regime, where the approximately linear behavior of the model is lost. Under this alternative scaling we prove that all fixed points of the dynamics in parameter space are global minimizers. We finally give examples of our convergence results in the case of models that diverge if trained with non-lazy TD learning, and in the case of neural networks.
accepted version to MSML 2021
References in corpus (18)
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- A Convergence Theory for Deep Learning via Over-Parameterization
- Gradient Descent Provably Optimizes Over-parameterized Neural Networks
- On Lazy Training in Differentiable Programming
- Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
- A Theoretical Analysis of Deep Q-Learning
- Neural Policy Gradient Methods: Global Optimality and Rates of Convergence
- Towards moderate overparameterization: global convergence guarantees for training shallow neural networks
- Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel
- Towards Characterizing Divergence in Deep Q-Learning
- A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth
- Approximate Temporal Difference Learning is a Gradient Descent for Reversible Policies
- Global convergence of neuron birth-death dynamics
- Single-Timescale Stochastic Nonconvex-Concave Optimization for Smooth Nonlinear TD Learning
- Neural Temporal-Difference and Q-Learning Provably Converge to Global Optima
- On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regime
- Geometric Insights into the Convergence of Nonlinear TD Learning
- On Lipschitz maps and the Hölder regularity of flows