Generalization in Deep Networks: The Role of Distance from Initialization

Abstract

Why does training deep neural networks using stochastic gradient descent (SGD) result in a generalization error that does not worsen with the number of parameters in the network? To answer this question, we advocate a notion of effective model capacity that is dependent on {\em a given random initialization of the network} and not just the training algorithm and the data distribution. We provide empirical evidences that demonstrate that the model capacity of SGD-trained deep networks is in fact restricted through implicit regularization of {\em the ${\ell }_2$ distance from the initialization}. We also provide theoretical arguments that further highlight the need for initialization-dependent notions of model capacity. We leave as open questions how and why distance from initialization is regularized, and whether it is sufficient to explain generalization.

arXiv commentSpotlight paper at NeurIPS 2017 workshop on Deep Learning: Bridging Theory and Practice

References in corpus (3)

• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• A Closer Look at Memorization in Deep Networks
• Sharp Minima Can Generalize For Deep Nets

Cited by in corpus (29)

• Fantastic Generalization Measures and Where to Find Them
• Review: Deep Learning in Electron Microscopy
• On Warm-Starting Neural Network Training
• Understanding the Failure Modes of Out-of-Distribution Generalization
• Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
• The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent
• How does Weight Correlation Affect the Generalisation Ability of Deep Neural Networks
• Assessing Generalization of SGD via Disagreement
• A Gradient Flow Framework For Analyzing Network Pruning
• Observational Overfitting in Reinforcement Learning
• Simple and Effective Regularization Methods for Training on Noisily Labeled Data with Generalization Guarantee
• Quantifying the generalization error in deep learning in terms of data distribution and neural network smoothness
• Training Algorithm Matters for the Performance of Neural Network Potential: A Case Study of Adam and the Kalman Filter Optimizers
• The intriguing role of module criticality in the generalization of deep networks
• AdjointNet: Constraining machine learning models with physics-based codes
• Robustness to Pruning Predicts Generalization in Deep Neural Networks
• Neural Complexity Measures
• Measuring Generalization with Optimal Transport
• A Theoretical Analysis of Fine-tuning with Linear Teachers
• Implicit Gradient Regularization
• Cockpit: A Practical Debugging Tool for the Training of Deep Neural Networks
• Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descent
• Improved generalization by noise enhancement
• LNPT: Label-free Network Pruning and Training
• Practical Assessment of Generalization Performance Robustness for Deep Networks via Contrastive Examples
• Extreme Memorization via Scale of Initialization
• Explaining generalization in deep learning: progress and fundamental limits
• Learning and Generalization in Overparameterized Normalizing Flows
• Understanding the Role of Adversarial Regularization in Supervised Learning