Pathological spectra of the Fisher information metric and its variants in deep neural networks

Abstract

The Fisher information matrix (FIM) plays an essential role in statistics and machine learning as a Riemannian metric tensor or a component of the Hessian matrix of loss functions. Focusing on the FIM and its variants in deep neural networks (DNNs), we reveal their characteristic scale dependence on the network width, depth and sample size when the network has random weights and is sufficiently wide. This study covers two widely-used FIMs for regression with linear output and for classification with softmax output. Both FIMs asymptotically show pathological eigenvalue spectra in the sense that a small number of eigenvalues become large outliers depending the width or sample size while the others are much smaller. It implies that the local shape of the parameter space or loss landscape is very sharp in a few specific directions while almost flat in the other directions. In particular, the softmax output disperses the outliers and makes a tail of the eigenvalue density spread from the bulk. We also show that pathological spectra appear in other variants of FIMs: one is the neural tangent kernel; another is a metric for the input signal and feature space that arises from feedforward signal propagation. Thus, we provide a unified perspective on the FIM and its variants that will lead to more quantitative understanding of learning in large-scale DNNs.

arXiv comment23 pages, 7 figures; v2: minor improvements, Section 3.4 added

References in corpus (21)

• Overcoming catastrophic forgetting in neural networks
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Deep Neural Networks as Gaussian Processes
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• On Exact Computation with an Infinitely Wide Neural Net
• Exponential expressivity in deep neural networks through transient chaos
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• Fisher-Rao Metric, Geometry, and Complexity of Neural Networks
• An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
• The Emergence of Spectral Universality in Deep Networks
• The large learning rate phase of deep learning: the catapult mechanism
• A Mean Field Theory of Batch Normalization
• Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach
• A Fine-Grained Spectral Perspective on Neural Networks
• Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians
• Fisher Information and Natural Gradient Learning of Random Deep Networks
• Order and Chaos: NTK views on DNN Normalization, Checkerboard and Boundary Artifacts
• The Normalization Method for Alleviating Pathological Sharpness in Wide Neural Networks

Cited by in corpus (7)

• Automatic Differentiable Monte Carlo: Theory and Application
• A Study of Gradient Variance in Deep Learning
• Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networks
• Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra
• Implicit Regularization via Neural Feature Alignment
• A Trace-restricted Kronecker-Factored Approximation to Natural Gradient
• Training Efficiency and Robustness in Deep Learning