Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training
arXiv:2101.12699 · doi:10.1073/pnas.2103091118
Abstract
In this paper, we introduce the \textit{Layer-Peeled Model}, a nonconvex yet analytically tractable optimization program, in a quest to better understand deep neural networks that are trained for a sufficiently long time. As the name suggests, this new model is derived by isolating the topmost layer from the remainder of the neural network, followed by imposing certain constraints separately on the two parts of the network. We demonstrate that the Layer-Peeled Model, albeit simple, inherits many characteristics of well-trained neural networks, thereby offering an effective tool for explaining and predicting common empirical patterns of deep learning training. First, when working on class-balanced datasets, we prove that any solution to this model forms a simplex equiangular tight frame, which in part explains the recently discovered phenomenon of neural collapse \cite{papyan2020prevalence}. More importantly, when moving to the imbalanced case, our analysis of the Layer-Peeled Model reveals a hitherto unknown phenomenon that we term \textit{Minority Collapse}, which fundamentally limits the performance of deep learning models on the minority classes. In addition, we use the Layer-Peeled Model to gain insights into how to mitigate Minority Collapse. Interestingly, this phenomenon is first predicted by the Layer-Peeled Model before being confirmed by our computational experiments.
Accepted at Proceedings of the National Academy of Sciences (PNAS); Changed the title
References in corpus (26)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- A Simple Framework for Contrastive Learning of Visual Representations
- A systematic study of the class imbalance problem in convolutional neural networks
- Reconciling modern machine learning practice and the bias-variance trade-off
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- A Convergence Theory for Deep Learning via Over-Parameterization
- Benign Overfitting in Linear Regression
- A Mean Field View of the Landscape of Two-Layers Neural Networks
- Meta-Weight-Net: Learning an Explicit Mapping For Sample Weighting
- Prevalence of Neural Collapse during the terminal phase of deep learning training
- Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss
- A Theoretical Analysis of Contrastive Unsupervised Representation Learning
- Gradient Descent Finds Global Minima of Deep Neural Networks
- Spectrally-normalized margin bounds for neural networks
- Convex Sparse Matrix Factorizations
- Just Interpolate: Kernel "Ridgeless" Regression Can Generalize
- A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics
- Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction
- A Selective Overview of Deep Learning
- Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets
- Mathematical Models of Overparameterized Neural Networks
- Recent advances in deep learning theory
- The Local Elasticity of Neural Networks
- Explicit regularization and implicit bias in deep network classifiers trained with the square loss