The Impact of Reinitialization on Generalization in Convolutional Neural Networks
arXiv:2109.00267
Abstract
Recent results suggest that reinitializing a subset of the parameters of a neural network during training can improve generalization, particularly for small training sets. We study the impact of different reinitialization methods in several convolutional architectures across 12 benchmark image classification datasets, analyzing their potential gains and highlighting limitations. We also introduce a new layerwise reinitialization algorithm that outperforms previous methods and suggest explanations of the observed improved generalization. First, we show that layerwise reinitialization increases the margin on the training examples without increasing the norm of the weights, hence leading to an improvement in margin-based generalization bounds for neural networks. Second, we demonstrate that it settles in flatter local minima of the loss surface. Third, it encourages learning general rules and discourages memorization by placing emphasis on the lower layers of the neural network. Our takeaway message is that the accuracy of convolutional neural networks can be improved for small datasets using bottom-up layerwise reinitialization, where the number of reinitialized layers may vary depending on the available compute budget.
arXiv comment12 figures, 7 tables
References in corpus (13)
- Scikit-learn: Machine Learning in Python
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- How transferable are features in deep neural networks?
- Layer Normalization
- DSD: Dense-Sparse-Dense Training for Deep Neural Networks
- iCassava 2019 Fine-Grained Visual Categorization Challenge
- What Do Neural Networks Learn When Trained With Random Labels?
- Deep Learning Through the Lens of Example Difficulty
- DNN or k-NN: That is the Generalize vs. Memorize Question
- RIFLE: Backpropagation in Depth for Deep Transfer Learning through Re-Initializing the Fully-connected LayEr