Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularization
arXiv:2108.11371
Abstract
Adaptive gradient methods such as Adam have gained increasing popularity in deep learning optimization. However, it has been observed that compared with (stochastic) gradient descent, Adam can converge to a different solution with a significantly worse test error in many deep learning applications such as image classification, even with a fine-tuned regularization. In this paper, we provide a theoretical explanation for this phenomenon: we show that in the nonconvex setting of learning over-parameterized two-layer convolutional neural networks starting from the same random initialization, for a class of data distributions (inspired from image data), Adam and gradient descent (GD) can converge to different global solutions of the training objective with provably different generalization errors, even with weight decay regularization. In contrast, we show that if the training objective is convex, and the weight decay regularization is employed, any optimization algorithms including Adam and GD will converge to the same solution if the training is successful. This suggests that the inferior generalization performance of Adam is fundamentally tied to the nonconvex landscape of deep learning optimization.
arXiv comment42 pages, 2 figures and 1 table
References in corpus (8)
- On the Convergence of Adam and Beyond
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- A Convergence Theory for Deep Learning via Over-Parameterization
- Improving Generalization Performance by Switching from Adam to SGD
- Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
- Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
- Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
- Analyzing Tensor Power Method Dynamics in Overcomplete Regime