A Recipe for Global Convergence Guarantee in Deep Neural Networks
arXiv:2104.05785
Abstract
Existing global convergence guarantees of (stochastic) gradient descent do not apply to practical deep networks in the practical regime of deep learning beyond the neural tangent kernel (NTK) regime. This paper proposes an algorithm, which is ensured to have global convergence guarantees in the practical regime beyond the NTK regime, under a verifiable condition called the expressivity condition. The expressivity condition is defined to be both data-dependent and architecture-dependent, which is the key property that makes our results applicable for practical settings beyond the NTK regime. On the one hand, the expressivity condition is theoretically proven to hold data-independently for fully-connected deep neural networks with narrow hidden layers and a single wide layer. On the other hand, the expressivity condition is numerically shown to hold data-dependently for deep (convolutional) ResNet with batch normalization with various standard image datasets. We also show that the proposed algorithm has generalization performances comparable with those of the heuristic algorithm, with the same hyper-parameters and total number of iterations. Therefore, the proposed algorithm can be viewed as a step towards providing theoretical guarantees for deep learning in the practical regime.
arXiv commentPublished in AAAI 2021
References in corpus (20)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- Adaptive activation functions accelerate convergence in deep and physics-informed neural networks
- A Mean Field View of the Landscape of Two-Layers Neural Networks
- Prevalence of Neural Collapse during the terminal phase of deep learning training
- Gradient Descent Provably Optimizes Over-parameterized Neural Networks
- Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
- Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
- On Lazy Training in Differentiable Programming
- Deep Equilibrium Models
- No bad local minima: Data independent training error guarantees for multilayer neural networks
- Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit
- Depth with Nonlinearity Creates No Bad Local Minima in ResNets
- Theory of Deep Learning III: explaining the non-overfitting puzzle
- Effect of Depth and Width on Local Minima in Deep Learning
- Ordered SGD: A New Stochastic Optimization Framework for Empirical Risk Minimization
- Network size and weights size for memorization with two-layers neural networks
- Neural Networks Learning and Memorization with (almost) no Over-Parameterization
- A Corrective View of Neural Networks: Representation, Memorization and Learning
- On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layers