An Unconstrained Layer-Peeled Perspective on Neural Collapse
arXiv:2110.02796
Abstract
Neural collapse is a highly symmetric geometric pattern of neural networks that emerges during the terminal phase of training, with profound implications on the generalization performance and robustness of the trained networks. To understand how the last-layer features and classifiers exhibit this recently discovered implicit bias, in this paper, we introduce a surrogate model called the unconstrained layer-peeled model (ULPM). We prove that gradient flow on this model converges to critical points of a minimum-norm separation problem exhibiting neural collapse in its global minimizer. Moreover, we show that the ULPM with the cross-entropy loss has a benign global landscape for its loss function, which allows us to prove that all the critical points are strict saddle points except the global minimizers that exhibit the neural collapse phenomenon. Empirically, we show that our results also hold during the training of neural networks in real-world tasks when explicit regularization or weight decay is not used.
Published as a conference paper at ICLR 2022
References in corpus (10)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Language Models are Few-Shot Learners
- Prevalence of Neural Collapse during the terminal phase of deep learning training
- A Geometric Analysis of Neural Collapse with Unconstrained Features
- An Exponential Learning Rate Schedule for Deep Learning
- Neural Collapse with Cross-Entropy Loss
- Explicit regularization and implicit bias in deep network classifiers trained with the square loss
- Analysis of the Optimization Landscapes for Overcomplete Representation Learning
- Gradient descent follows the regularization path for general losses