Limitations of the Empirical Fisher Approximation for Natural Gradient Descent

Abstract

Natural gradient descent, which preconditions a gradient descent update with the Fisher information matrix of the underlying statistical model, is a way to capture partial second-order information. Several highly visible works have advocated an approximation known as the empirical Fisher, drawing connections between approximate second-order methods and heuristics like Adam. We dispute this argument by showing that the empirical Fisher---unlike the Fisher---does not generally capture second-order information. We further argue that the conditions under which the empirical Fisher approaches the Fisher (and the Hessian) are unlikely to be met in practice, and that, even on simple optimization problems, the pathologies of the empirical Fisher can have undesirable effects.

arXiv commentV3: Minor corrections (typographic errors)

Cited by in corpus (29)

• WoodFisher: Efficient Second-Order Approximation for Neural Network Compression
• Scalable Second Order Optimization for Deep Learning
• Laplace Redux -- Effortless Bayesian Deep Learning
• Quantum Continual Learning Overcoming Catastrophic Forgetting
• A Study of Gradient Variance in Deep Learning
• Estimating Model Uncertainty of Neural Networks in Sparse Information Form
• Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning
• Information-geometry of physics-informed statistical manifolds and its use in data assimilation
• Optimization of Graph Neural Networks with Natural Gradient Descent
• M-FAC: Efficient Matrix-Free Approximations of Second-Order Information
• Natural continual learning: success is a journey, not (just) a destination
• Incremental Learning for Personalized Recommender Systems
• Interactive Label Cleaning with Example-based Explanations
• Does the Data Induce Capacity Control in Deep Learning?
• An Information-Geometric Distance on the Space of Tasks
• Disentangling the Gauss-Newton Method and Approximate Inference for Neural Networks
• Two-Level K-FAC Preconditioning for Deep Learning
• L2M: Practical posterior Laplace approximation with optimization-driven second moment estimation
• Predictive coding, precision and natural gradients
• ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure
• Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks
• TENGraD: Time-Efficient Natural Gradient Descent with Exact Fisher-Block Inversion
• Stable Anderson Acceleration for Deep Learning
• PredProp: Bidirectional Stochastic Optimization with Precision Weighted Predictive Coding
• Training Efficiency and Robustness in Deep Learning
• Vanishing Curvature and the Power of Adaptive Methods in Randomly Initialized Deep Networks
• Variational Laplace for Bayesian neural networks
• On the Variance of the Fisher Information for Deep Learning
• Tensor Normal Training for Deep Learning Models