Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians

Abstract

We consider deep classifying neural networks. We expose a structure in the derivative of the logits with respect to the parameters of the model, which is used to explain the existence of outliers in the spectrum of the Hessian. Previous works decomposed the Hessian into two components, attributing the outliers to one of them, the so-called Covariance of gradients. We show this term is not a Covariance but a second moment matrix, i.e., it is influenced by means of gradients. These means possess an additive two-way structure that is the source of the outliers in the spectrum. This structure can be used to approximate the principal subspace of the Hessian using certain "averaging" operations, avoiding the need for high-dimensional eigenanalysis. We corroborate this claim across different datasets, architectures and sample sizes.

References in corpus (4)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Gradient Descent Happens in a Tiny Subspace

Cited by in corpus (17)

• Prevalence of Neural Collapse during the terminal phase of deep learning training
• Asymptotics of Wide Networks from Feynman Diagrams
• The Break-Even Point on Optimization Trajectories of Deep Neural Networks
• Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identification
• Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networks
• Directional Pruning of Deep Neural Networks
• The asymptotic spectrum of the Hessian of DNN throughout training
• Beyond Random Matrix Theory for Deep Networks
• Analytic Insights into Structure and Rank of Neural Network Hessian Maps
• De-randomized PAC-Bayes Margin Bounds: Applications to Non-convex and Non-smooth Predictors
• Universal characteristics of deep neural network loss surfaces from random matrix theory
• Does the Data Induce Capacity Control in Deep Learning?
• How many degrees of freedom do we need to train deep networks: a loss landscape perspective
• Appearance of Random Matrix Theory in Deep Learning
• ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure
• A Framework for Private Matrix Analysis
• Dynamic Game Theoretic Neural Optimizer