What training reveals about neural network complexity

Abstract

This work explores the Benevolent Training Hypothesis (BTH) which argues that the complexity of the function a deep neural network (NN) is learning can be deduced by its training dynamics. Our analysis provides evidence for BTH by relating the NN's Lipschitz constant at different regions of the input space with the behavior of the stochastic training procedure. We first observe that the Lipschitz constant close to the training data affects various aspects of the parameter trajectory, with more complex networks having a longer trajectory, bigger variance, and often veering further from their initialization. We then show that NNs whose 1st layer bias is trained more steadily (i.e., slowly and with little variation) have bounded complexity even in regions of the input space that are far from any training point. Finally, we find that steady training with Dropout implies a training- and data-dependent generalization bound that grows poly-logarithmically with the number of parameters. Overall, our results support the intuition that good training behavior can be a useful bias towards good generalization.

arXiv commentPublished at NeurIPS 2021

References in corpus (21)

• Improving neural networks by preventing co-adaptation of feature detectors
• Understanding deep learning requires rethinking generalization
• On the Number of Linear Regions of Deep Neural Networks
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• Robust Large Margin Deep Neural Networks
• Three Factors Influencing Minima in SGD
• Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?
• On the Spectral Bias of Neural Networks
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• Sharp Minima Can Generalize For Deep Nets
• A PAC-Bayesian Tutorial with A Dropout Bound
• Data-Dependent Stability of Stochastic Gradient Descent
• Complexity of Linear Regions in Deep Networks
• Theory of Deep Learning IIb: Optimization Properties of SGD
• Lipschitz constant estimation of Neural Networks via sparse polynomial optimization
• Topology and Geometry of Half-Rectified Network Optimization
• What graph neural networks cannot learn: depth vs width
• Analysis of the Gradient Descent Algorithm for a Deep Neural Network Model with Skip-connections
• Dropout: Explicit Forms and Capacity Control
• Lipschitz Networks and Distributional Robustness