ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure

Abstract

Curvature in form of the Hessian or its generalized Gauss-Newton (GGN) approximation is valuable for algorithms that rely on a local model for the loss to train, compress, or explain deep networks. Existing methods based on implicit multiplication via automatic differentiation or Kronecker-factored block diagonal approximations do not consider noise in the mini-batch. We present ViViT, a curvature model that leverages the GGN's low-rank structure without further approximations. It allows for efficient computation of eigenvalues, eigenvectors, as well as per-sample first- and second-order directional derivatives. The representation is computed in parallel with gradients in one backward pass and offers a fine-grained cost-accuracy trade-off, which allows it to scale. We demonstrate this by conducting performance benchmarks and substantiate ViViT's usefulness by studying the impact of noise on the GGN's structural properties during neural network training.

arXiv commentMain text: 10 pages, 6 figures; Supplements: 26 pages, 27 figures, 5 tables

References in corpus (18)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• New insights and perspectives on the natural gradient method
• Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting
• Gradient Descent Happens in a Tiny Subspace
• Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
• An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
• The Early Phase of Neural Network Training
• Limitations of the Empirical Fisher Approximation for Natural Gradient Descent
• Practical Gauss-Newton Optimisation for Deep Learning
• PyHessian: Neural Networks Through the Lens of the Hessian
• The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size
• Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians
• Estimating the Spectral Density of Large Implicit Matrices
• Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalization
• On the Promise of the Stochastic Generalized Gauss-Newton Method for Training DNNs
• Cockpit: A Practical Debugging Tool for the Training of Deep Neural Networks
• Deep Curvature Suite