Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra
arXiv:2008.11865
Abstract
Numerous researchers recently applied empirical spectral analysis to the study of modern deep learning classifiers. We identify and discuss an important formal class/cross-class structure and show how it lies at the origin of the many visually striking features observed in deepnet spectra, some of which were reported in recent articles, others are unveiled here for the first time. These include spectral outliers, "spikes", and small but distinct continuous distributions, "bumps", often seen beyond the edge of a "main bulk". The significance of the cross-class structure is illustrated in three ways: (i) we prove the ratio of outliers to bulk in the spectrum of the Fisher information matrix is predictive of misclassification, in the context of multinomial logistic regression; (ii) we demonstrate how, gradually with depth, a network is able to separate class-distinctive information from class variability, all while orthogonalizing the class-distinctive information; and (iii) we propose a correction to KFAC, a well-known second-order optimization algorithm for training deepnets.
References in corpus (27)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
- Manifold Mixup: Better Representations by Interpolating Hidden States
- Three Factors Influencing Minima in SGD
- Scaling description of generalization with number of parameters in deep learning
- Fantastic Generalization Measures and Where to Find Them
- Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
- Gradient Descent Happens in a Tiny Subspace
- Are All Layers Created Equal?
- Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
- An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
- Measuring the Intrinsic Dimension of Objective Landscapes
- The large learning rate phase of deep learning: the catapult mechanism
- On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length
- Asymptotics of Wide Networks from Feynman Diagrams
- Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobian
- EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis
- PyHessian: Neural Networks Through the Lens of the Hessian
- Hessian-Aware Zeroth-Order Optimization for Black-Box Adversarial Attack
- Negative eigenvalues of the Hessian in deep neural networks
- Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics
- Emergent properties of the local geometry of neural loss landscapes
- Asymptotics of Wide Convolutional Neural Networks
- Pathological spectra of the Fisher information metric and its variants in deep neural networks
- Hessian based analysis of SGD for Deep Nets: Dynamics and Generalization