Prevalence of Neural Collapse during the terminal phase of deep learning training

Abstract

Modern practice for training classification deepnets involves a Terminal Phase of Training (TPT), which begins at the epoch where training error first vanishes; During TPT, the training error stays effectively zero while training loss is pushed towards zero. Direct measurements of TPT, for three prototypical deepnet architectures and across seven canonical classification datasets, expose a pervasive inductive bias we call Neural Collapse, involving four deeply interconnected phenomena: (NC1) Cross-example within-class variability of last-layer training activations collapses to zero, as the individual activations themselves collapse to their class-means; (NC2) The class-means collapse to the vertices of a Simplex Equiangular Tight Frame (ETF); (NC3) Up to rescaling, the last-layer classifiers collapse to the class-means, or in other words to the Simplex ETF, i.e. to a self-dual configuration; (NC4) For a given activation, the classifier's decision collapses to simply choosing whichever class has the closest train class-mean, i.e. the Nearest Class Center (NCC) decision rule. The symmetric and very simple geometry induced by the TPT confers important benefits, including better generalization performance, better robustness, and better interpretability.

References in corpus (17)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Reconciling modern machine learning practice and the bias-variance trade-off
• Parseval Networks: Improving Robustness to Adversarial Examples
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolate
• The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning
• Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
• Memory-Efficient Implementation of DenseNets
• Does data interpolation contradict statistical optimality?
• Convolutional Neural Networks Analyzed via Convolutional Sparse Coding
• Identifying Mislabeled Instances in Classification Datasets
• Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians
• Discrete Deep Feature Extraction: A Theory and New Architectures
• Deep Network Classification by Scattering and Homotopy Dictionary Learning
• When and How Can Deep Generative Models be Inverted?

Cited by in corpus (50)

• A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part II: Applications, Cognitive Models, and Challenges
• Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training
• ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reduction
• A Geometric Analysis of Neural Collapse with Unconstrained Features
• Neural Collapse Inspired Attraction-Repulsion-Balanced Loss for Imbalanced Learning
• Mathematical Models of Overparameterized Neural Networks
• Deep regularization and direct training of the inner layers of Neural Networks with Kernel Flows
• Understanding Deep Learning via Decision Boundary
• Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path
• Perceptron Theory Can Predict the Accuracy of Neural Networks
• Closed-Loop Data Transcription to an LDR via Minimaxing Rate Reduction
• Gaussian Universality of Perceptrons with Random Labels
• Prototypical Partial Optimal Transport for Universal Domain Adaptation
• Revealing the Structure of Deep Neural Networks via Convex Duality
• Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolation
• Toward Better Generalization Bounds with Locally Elastic Stability
• Neural Collapse with Cross-Entropy Loss
• Federated Optimization of Smooth Loss Functions
• An Unconstrained Layer-Peeled Perspective on Neural Collapse
• E${}^2$CM: Early Exit via Class Means for Efficient Supervised and Unsupervised Learning
• Label-Imbalanced and Group-Sensitive Classification under Overparameterization
• Demystifying Batch Normalization in ReLU Networks: Equivalent Convex Optimization Models and Implicit Regularization
• Coding schemes in neural networks learning classification tasks
• Explicit regularization and implicit bias in deep network classifiers trained with the square loss
• CoReS: Compatible Representations via Stationarity
• Separation and Concentration in Deep Networks
• Learning to Assimilate in Chaotic Dynamical Systems
• A Recipe for Global Convergence Guarantee in Deep Neural Networks
• Learn by Reasoning: Analogical Weight Generation for Few-Shot Class-Incremental Learning
• Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensions
• Distribution of Classification Margins: Are All Data Equal?
• Dual-Head Knowledge Distillation: Enhancing Logits Utilization with an Auxiliary Head
• Theoretical Guarantees for Low-Rank Compression of Deep Neural Networks
• Sample-aware RandAugment: Search-free Automatic Data Augmentation for Effective Image Recognition
• Evaluating the Fairness of Neural Collapse in Medical Image Classification
• Dissecting Supervised Contrastive Learning
• Probing neural networks with t-SNE, class-specific projections and a guided tour
• Double descent in quantum kernel methods
• Properties of the After Kernel
• Dependence Induced Representations
• Why Do Better Loss Functions Lead to Less Transferable Features?
• Imitating Deep Learning Dynamics via Locally Elastic Stochastic Differential Equations
• Incremental Learning via Rate Reduction
• Unsupervised Learning of Unbiased Visual Representations
• Gradient flow in parameter space is equivalent to linear interpolation in output space
• Exploring the high dimensional geometry of HSI features
• Deep Minimax Classifiers for Imbalanced Datasets with a Small Number of Minority Samples
• Rethinking the Backbone in Class Imbalanced Federated Source Free Domain Adaptation: The Utility of Vision Foundation Models
• Reviewing Model Collapse and Countermeasures
• Learning to Give Checkable Answers with Prover-Verifier Games