A Geometric Analysis of Neural Collapse with Unconstrained Features

Abstract

We provide the first global optimization landscape analysis of $NeuralCollapse$ -- an intriguing empirical phenomenon that arises in the last-layer classifiers and features of neural networks during the terminal phase of training. As recently reported by Papyan et al., this phenomenon implies that ($i$) the class means and the last-layer classifiers all collapse to the vertices of a Simplex Equiangular Tight Frame (ETF) up to scaling, and ($ii$) cross-example within-class variability of last-layer activations collapses to zero. We study the problem based on a simplified $unconstrainedfeaturemodel$, which isolates the topmost layers from the classifier of the neural network. In this context, we show that the classical cross-entropy loss with weight decay has a benign global landscape, in the sense that the only global minimizers are the Simplex ETFs while all other critical points are strict saddles whose Hessian exhibit negative curvature directions. In contrast to existing landscape analysis for deep neural networks which is often disconnected from practice, our analysis of the simplified model not only does it explain what kind of features are learned in the last layer, but it also shows why they can be efficiently optimized in the simplified settings, matching the empirical observations in practical deep network architectures. These findings could have profound implications for optimization, generalization, and robustness of broad interests. For example, our experiments demonstrate that one may set the feature dimension equal to the number of classes and fix the last-layer classifier to be a Simplex ETF for network training, which reduces memory cost by over $20\%$ on ResNet18 without sacrificing the generalization performance.

arXiv comment42 pages, 8 figures, 1 table; the first two authors contributed to this work equally

References in corpus (45)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• A Simple Framework for Contrastive Learning of Visual Representations
• Reconciling modern machine learning practice and the bias-variance trade-off
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Benign Overfitting in Linear Regression
• Prevalence of Neural Collapse during the terminal phase of deep learning training
• On Lazy Training in Differentiable Programming
• Complete Dictionary Recovery over the Sphere I: Overview and the Geometric Picture
• Global Optimality of Local Search for Low Rank Matrix Recovery
• The generalization error of random features regression: Precise asymptotics and double descent curve
• No Spurious Local Minima in Nonconvex Low Rank Problems: A Unified Geometric Analysis
• Provable approximation properties for deep neural networks
• Matrix Completion has No Spurious Local Minimum
• Characterizing Implicit Bias in Terms of Optimization Geometry
• Spurious Local Minima are Common in Two-Layer ReLU Neural Networks
• Global Optimality in Low-rank Matrix Optimization
• Complete Dictionary Recovery over the Sphere II: Recovery by Riemannian Trust-region Method
• The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning
• Deep Residual Networks with Exponential Linear Unit
• When Are Nonconvex Problems Not Scary?
• Gradient Descent Maximizes the Margin of Homogeneous Neural Networks
• Pre-training Tasks for Embedding-based Large-scale Retrieval
• Global Optimality in Tensor Factorization, Deep Learning, and Beyond
• The Non-convex Geometry of Low-rank Matrix Optimization
• The Global Landscape of Neural Networks: An Overview
• Implicit Regularization in Deep Learning May Not Be Explainable by Norms
• Intriguing Properties of Contrastive Losses
• Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction
• Adding One Neuron Can Eliminate All Bad Local Minima
• Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks
• Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Rate
• From Symmetry to Geometry: Tractable Nonconvex Problems
• Small nonlinearities in activation functions create bad local minima in neural networks
• Deep Isometric Learning for Visual Recognition
• Neural Collapse with Cross-Entropy Loss
• Deep Networks from the Principle of Rate Reduction
• Finding the Sparsest Vectors in a Subspace: Theory, Algorithms, and Applications
• A Local Convergence Theory for Mildly Over-Parameterized Two-Layer Neural Network
• Inductive Bias of Multi-Channel Linear Convolutional Networks with Bounded Weight Norm
• Benign Overfitting and Noisy Features
• Gradient descent follows the regularization path for general losses
• The Implicit Bias of AdaGrad on Separable Data

Cited by in corpus (6)

• ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reduction
• Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path
• An Unconstrained Layer-Peeled Perspective on Neural Collapse
• CoReS: Compatible Representations via Stationarity
• Identifying Critical Neurons in ANN Architectures using Mixed Integer Programming
• Dependence Induced Representations