Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians

Abstract

We consider deep classifying neural networks. We expose a structure in the derivative of the logits with respect to the parameters of the model, which is used to explain the existence of outliers in the spectrum of the Hessian. Previous works decomposed the Hessian into two components, attributing the outliers to one of them, the so-called Covariance of gradients. We show this term is not a Covariance but a second moment matrix, i.e., it is influenced by means of gradients. These means possess an additive two-way structure that is the source of the outliers in the spectrum. This structure can be used to approximate the principal subspace of the Hessian using certain "averaging" operations, avoiding the need for high-dimensional eigenanalysis. We corroborate this claim across different datasets, architectures and sample sizes.

References in corpus (9)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• The jamming transition as a paradigm to understand the loss landscape of deep neural networks
• Gradient Descent Happens in a Tiny Subspace
• Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• Fluctuation-dissipation relations for stochastic gradient descent

Cited by in corpus (21)

• Prevalence of Neural Collapse during the terminal phase of deep learning training
• Stiffness: A New Perspective on Generalization in Neural Networks
• Asymptotics of Wide Networks from Feynman Diagrams
• The Break-Even Point on Optimization Trajectories of Deep Neural Networks
• The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size
• Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identification
• Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networks
• Directional Pruning of Deep Neural Networks
• Pathological spectra of the Fisher information metric and its variants in deep neural networks
• The asymptotic spectrum of the Hessian of DNN throughout training
• Beyond Random Matrix Theory for Deep Networks
• Hessian based analysis of SGD for Deep Nets: Dynamics and Generalization
• Analytic Insights into Structure and Rank of Neural Network Hessian Maps
• De-randomized PAC-Bayes Margin Bounds: Applications to Non-convex and Non-smooth Predictors
• Universal characteristics of deep neural network loss surfaces from random matrix theory
• Does the Data Induce Capacity Control in Deep Learning?
• How many degrees of freedom do we need to train deep networks: a loss landscape perspective
• A Framework for Private Matrix Analysis
• ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure
• Appearance of Random Matrix Theory in Deep Learning
• Dynamic Game Theoretic Neural Optimizer