Trap of Feature Diversity in the Learning of MLPs
arXiv:2112.00980
Abstract
In this paper, we focus on a typical two-phase phenomenon in the learning of multi-layer perceptrons (MLPs), and we aim to explain the reason for the decrease of feature diversity in the first phase. Specifically, people find that, in the training of MLPs, the training loss does not decrease significantly until the second phase. To this end, we further explore the reason why the diversity of features over different samples keeps decreasing in the first phase, which hurts the optimization of MLPs. We explain such a phenomenon in terms of the learning dynamics of MLPs. Furthermore, we theoretically explain why four typical operations can alleviate the decrease of the feature diversity.
References in corpus (21)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
- Opening the Black Box of Deep Neural Networks via Information
- The Loss Surfaces of Multilayer Networks
- Deep Neural Networks as Gaussian Processes
- A Closer Look at Memorization in Deep Networks
- Sensitivity and Generalization in Neural Networks: an Empirical Study
- Three Factors Influencing Minima in SGD
- Information-theoretic analysis of generalization capability of learning algorithms
- Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
- Spurious Local Minima are Common in Two-Layer ReLU Neural Networks
- Regularizing CNNs with Locally Constrained Decorrelations
- Stiffness: A New Perspective on Generalization in Neural Networks
- An analytic theory of generalization dynamics and transfer learning in deep linear networks
- Generalization bounds for deep convolutional neural networks
- Understanding training and generalization in deep learning by Fourier analysis
- The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalization
- Triple descent and the two kinds of overfitting: Where & why do they appear?
- A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks
- When Will Gradient Methods Converge to Max-margin Classifier under ReLU Models?