Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach

Abstract

The Fisher information matrix (FIM) is a fundamental quantity to represent the characteristics of a stochastic model, including deep neural networks (DNNs). The present study reveals novel statistics of FIM that are universal among a wide class of DNNs. To this end, we use random weights and large width limits, which enables us to utilize mean field theories. We investigate the asymptotic statistics of the FIM's eigenvalues and reveal that most of them are close to zero while the maximum eigenvalue takes a huge value. Because the landscape of the parameter space is defined by the FIM, it is locally flat in most dimensions, but strongly distorted in others. Moreover, we demonstrate the potential usage of the derived statistics in learning strategies. First, small eigenvalues that induce flatness can be connected to a norm-based capacity measure of generalization ability. Second, the maximum eigenvalue that induces the distortion enables us to quantitatively estimate an appropriately sized learning rate for gradient methods to converge.

Accepted at AISTATS2019. Main text: 10 pages, 2 figures. Supplementary material: 9 pages, 2 figures, typos corrected

References in corpus (19)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Overcoming catastrophic forgetting in neural networks
• On the Number of Linear Regions of Deep Neural Networks
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Deep Neural Networks as Gaussian Processes
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• Exponential expressivity in deep neural networks through transient chaos
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
• The loss surface of deep and wide neural networks
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• Sharp Minima Can Generalize For Deep Nets
• Fisher-Rao Metric, Geometry, and Complexity of Neural Networks
• The Emergence of Spectral Universality in Deep Networks
• Norm-Based Capacity Control in Neural Networks
• Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks
• The Landscape of Empirical Risk for Non-convex Losses
• Exponentially vanishing sub-optimal local minima in multilayer neural networks
• Exploring the Function Space of Deep-Learning Machines

Cited by in corpus (24)

• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Measuring the Effects of Data Parallelism on Neural Network Training
• Fisher-Rao Metric, Geometry, and Complexity of Neural Networks
• Neural Tangents: Fast and Easy Infinite Neural Networks in Python
• Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspective
• Finite Versus Infinite Neural Networks: an Empirical Study
• Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks
• Bayesian Deep Ensembles via the Neural Tangent Kernel
• Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networks
• Towards NNGP-guided Neural Architecture Search
• Using a thousand optimization tasks to learn hyperparameter search strategies
• When Does Preconditioning Help or Hurt Generalization?
• Order and Chaos: NTK views on DNN Normalization, Checkerboard and Boundary Artifacts
• Pathological spectra of the Fisher information metric and its variants in deep neural networks
• Information Geometry of Orthogonal Initializations and Training
• The asymptotic spectrum of the Hessian of DNN throughout training
• Does the Data Induce Capacity Control in Deep Learning?
• Any Target Function Exists in a Neighborhood of Any Sufficiently Wide Random Network: A Geometrical Perspective
• Towards Query-Efficient Black-Box Adversary with Zeroth-Order Natural Gradient Descent
• Analyzing Finite Neural Networks: Can We Trust Neural Tangent Kernel Theory?
• Approximate Spectral Decomposition of Fisher Information Matrix for Simple ReLU Networks
• Sparsification as a Remedy for Staleness in Distributed Asynchronous SGD
• Deep network as memory space: complexity, generalization, disentangled representation and interpretability
• Adversarial attacks on neural networks through canonical Riemannian foliations