Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks

Abstract

Adaptive gradient methods, which adopt historical gradient information to automatically adjust the learning rate, despite the nice property of fast convergence, have been observed to generalize worse than stochastic gradient descent (SGD) with momentum in training deep neural networks. This leaves how to close the generalization gap of adaptive gradient methods an open problem. In this work, we show that adaptive gradient methods such as Adam, Amsgrad, are sometimes "over adapted". We design a new algorithm, called Partially adaptive momentum estimation method, which unifies the Adam/Amsgrad with SGD by introducing a partial adaptive parameter $p$, to achieve the best from both worlds. We also prove the convergence rate of our proposed algorithm to a stationary point in the stochastic nonconvex optimization setting. Experiments on standard benchmarks show that our proposed algorithm can maintain a fast convergence rate as Adam/Amsgrad while generalizing as well as SGD in training deep neural networks. These results would suggest practitioners pick up adaptive gradient methods once again for faster training of deep neural networks.

arXiv comment17 pages, 4 figures, 4 tables. In IJCAI 2020

References in corpus (21)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Decoupled Weight Decay Regularization
• ADADELTA: An Adaptive Learning Rate Method
• Wide Residual Networks
• On the Convergence of Adam and Beyond
• The Marginal Value of Adaptive Gradient Methods in Machine Learning
• Improving Generalization Performance by Switching from Adam to SGD
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• Stochastic Variance Reduction for Nonconvex Optimization
• AdaGrad stepsizes: Sharp convergence over nonconvex landscapes
• Adaptive Bound Optimization for Online Convex Optimization
• Variants of RMSProp and Adagrad with Logarithmic Regret Bounds
• On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization
• Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networks
• Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov acceleration
• The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure For Least Squares
• A Walk with SGD
• Normalized Direction-preserving Adam
• Accelerate Stochastic Subgradient Method by Leveraging Local Growth Condition

Cited by in corpus (46)

• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• On the Variance of the Adaptive Learning Rate and Beyond
• Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks
• AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients
• On the Convergence Proof of AMSGrad and a New Version
• On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization
• Understanding the Role of Training Regimes in Continual Learning
• Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout
• Nostalgic Adam: Weighting more of the past gradients when designing the adaptive learning rate
• Adam revisited: a weighted past gradients perspective
• High Frequency Component Helps Explain the Generalization of Convolutional Neural Networks
• Global Convergence of Adaptive Gradient Methods for An Over-parameterized Neural Network
• Sequential Training of Neural Networks with Gradient Boosting
• Unitary long-time evolution with quantum renormalization groups and artificial neural networks
• AdaX: Adaptive Gradient Descent with Exponential Long Term Memory
• On the adequacy of untuned warmup for adaptive optimization
• Mixing ADAM and SGD: a Combined Optimization Method
• Effective Federated Adaptive Gradient Methods with Non-IID Decentralized Data
• LaProp: Separating Momentum and Adaptivity in Adam
• Optimizer Benchmarking Needs to Account for Hyperparameter Tuning
• A Second look at Exponential and Cosine Step Sizes: Simplicity, Adaptivity, and Performance
• Adam${}^{+}$: A Stochastic Method with Adaptive Variance Reduction
• Positive-Negative Momentum: Manipulating Stochastic Gradient Noise to Improve Generalization
• Disentangling Adaptive Gradient Methods from Learning Rates
• Convergence Analysis of a Momentum Algorithm with Adaptive Step Size for Non Convex Optimization
• BiAdam: Fast Adaptive Bilevel Optimization Methods
• Proximal Adam: Robust Adaptive Update Scheme for Constrained Optimization
• Training Neural Networks for and by Interpolation
• AdaSGD: Bridging the gap between SGD and Adam
• Why ResNet Works? Residuals Generalize
• A new regret analysis for Adam-type algorithms
• Effectiveness of Optimization Algorithms in Deep Image Classification
• Flatness is a False Friend
• The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networks
• Parallel and distributed asynchronous adaptive stochastic gradient methods
• On the Trend-corrected Variant of Adaptive Stochastic Optimization Methods
• Momentum Centering and Asynchronous Update for Adaptive Gradient Methods
• Blockwise Adaptivity: Faster Training and Better Generalization in Deep Learning
• Adaptive Gradient Method with Resilience and Momentum
• TAG: Task-based Accumulated Gradients for Lifelong learning
• A Comprehensive Study on Optimization Strategies for Gradient Descent In Deep Learning
• ACMo: Angle-Calibrated Moment Methods for Stochastic Optimization
• Granular Motor State Monitoring of Free Living Parkinson's Disease Patients via Deep Learning
• Extreme Tensoring for Low-Memory Preconditioning
• Rapidly Adapting Moment Estimation
• MaxVA: Fast Adaptation of Step Sizes by Maximizing Observed Variance of Gradients