A Trace-restricted Kronecker-Factored Approximation to Natural Gradient
arXiv:2011.10741
Abstract
Second-order optimization methods have the ability to accelerate convergence by modifying the gradient through the curvature matrix. There have been many attempts to use second-order optimization methods for training deep neural networks. Inspired by diagonal approximations and factored approximations such as Kronecker-Factored Approximate Curvature (KFAC), we propose a new approximation to the Fisher information matrix (FIM) called Trace-restricted Kronecker-factored Approximate Curvature (TKFAC) in this work, which can hold the certain trace relationship between the exact and the approximate FIM. In TKFAC, we decompose each block of the approximate FIM as a Kronecker product of two smaller matrices and scaled by a coefficient related to trace. We theoretically analyze TKFAC's approximation error and give an upper bound of it. We also propose a new damping technique for TKFAC on convolutional neural networks to maintain the superiority of second-order optimization methods during training. Experiments show that our method has better performance compared with several state-of-the-art algorithms on some deep network architectures.
References in corpus (9)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization
- High-dimensional covariance matrix estimation in approximate factor models
- New insights and perspectives on the natural gradient method
- Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis
- Practical Quasi-Newton Methods for Training Deep Neural Networks
- Training Neural Networks with Stochastic Hessian-Free Optimization
- Pathological spectra of the Fisher information metric and its variants in deep neural networks
- Eigenvalue Corrected Noisy Natural Gradient