An Unconstrained Layer-Peeled Perspective on Neural Collapse
arXiv:2110.02796
Abstract
Neural collapse is a highly symmetric geometric pattern of neural networks that emerges during the terminal phase of training, with profound implications on the generalization performance and robustness of the trained networks. To understand how the last-layer features and classifiers exhibit this recently discovered implicit bias, in this paper, we introduce a surrogate model called the unconstrained layer-peeled model (ULPM). We prove that gradient flow on this model converges to critical points of a minimum-norm separation problem exhibiting neural collapse in its global minimizer. Moreover, we show that the ULPM with the cross-entropy loss has a benign global landscape for its loss function, which allows us to prove that all the critical points are strict saddle points except the global minimizers that exhibit the neural collapse phenomenon. Empirically, we show that our results also hold during the training of neural networks in real-world tasks when explicit regularization or weight decay is not used.
arXiv commentPublished as a conference paper at ICLR 2022
References in corpus (24)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Language Models are Few-Shot Learners
- Deep Potential Molecular Dynamics: a scalable model with the accuracy of quantum mechanics
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- Prevalence of Neural Collapse during the terminal phase of deep learning training
- Complete Dictionary Recovery over the Sphere I: Overview and the Geometric Picture
- Matrix Completion has No Spurious Local Minimum
- When Are Nonconvex Problems Not Scary?
- Gradient Descent Maximizes the Margin of Homogeneous Neural Networks
- Global Optimality in Tensor Factorization, Deep Learning, and Beyond
- Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training
- Implicit Regularization in Deep Learning May Not Be Explainable by Norms
- An Exponential Learning Rate Schedule for Deep Learning
- A Geometric Analysis of Neural Collapse with Unconstrained Features
- From Symmetry to Geometry: Tractable Nonconvex Problems
- Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networks
- Neural Collapse with Cross-Entropy Loss
- When Does Preconditioning Help or Hurt Generalization?
- Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theory
- Explicit regularization and implicit bias in deep network classifiers trained with the square loss
- To Each Optimizer a Norm, To Each Norm its Generalization
- Analysis of the Optimization Landscapes for Overcomplete Representation Learning
- Gradient descent follows the regularization path for general losses