A Recipe for Global Convergence Guarantee in Deep Neural Networks
arXiv:2104.05785
Abstract
Existing global convergence guarantees of (stochastic) gradient descent do not apply to practical deep networks in the practical regime of deep learning beyond the neural tangent kernel (NTK) regime. This paper proposes an algorithm, which is ensured to have global convergence guarantees in the practical regime beyond the NTK regime, under a verifiable condition called the expressivity condition. The expressivity condition is defined to be both data-dependent and architecture-dependent, which is the key property that makes our results applicable for practical settings beyond the NTK regime. On the one hand, the expressivity condition is theoretically proven to hold data-independently for fully-connected deep neural networks with narrow hidden layers and a single wide layer. On the other hand, the expressivity condition is numerically shown to hold data-dependently for deep (convolutional) ResNet with batch normalization with various standard image datasets. We also show that the proposed algorithm has generalization performances comparable with those of the heuristic algorithm, with the same hyper-parameters and total number of iterations. Therefore, the proposed algorithm can be viewed as a step towards providing theoretical guarantees for deep learning in the practical regime.
Published in AAAI 2021
References in corpus (8)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Prevalence of Neural Collapse during the terminal phase of deep learning training
- Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
- Deep Equilibrium Models
- Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit
- Theory of Deep Learning III: explaining the non-overfitting puzzle
- Neural Networks Learning and Memorization with (almost) no Over-Parameterization
- On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layers