Adaptive Gradient Methods with Dynamic Bound of Learning Rate

Abstract

Adaptive optimization methods such as AdaGrad, RMSprop and Adam have been proposed to achieve a rapid training process with an element-wise scaling term on learning rates. Though prevailing, they are observed to generalize poorly compared with SGD or even fail to converge due to unstable and extreme learning rates. Recent work has put forward some algorithms such as AMSGrad to tackle this issue but they failed to achieve considerable improvement over existing methods. In our paper, we demonstrate that extreme learning rates can lead to poor performance. We provide new variants of Adam and AMSGrad, called AdaBound and AMSBound respectively, which employ dynamic bounds on learning rates to achieve a gradual and smooth transition from adaptive methods to SGD and give a theoretical proof of convergence. We further conduct experiments on various popular tasks and models, which is often insufficient in previous work. Experimental results show that new variants can eliminate the generalization gap between adaptive methods and SGD and maintain higher learning speed early in training at the same time. Moreover, they can bring significant improvement over their prototypes, especially on complex deep networks. The implementation of the algorithm can be found at https://github.com/Luolc/AdaBound .

Accepted to ICLR 2019. arXiv admin note: text overlap with arXiv:1904.09237 by other authors

Cited by in corpus (69)

• On the Variance of the Adaptive Learning Rate and Beyond
• FedALA: Adaptive Local Aggregation for Personalized Federated Learning
• How Does Learning Rate Decay Help Modern Neural Networks?
• Adaptive Federated Optimization
• Optimize TSK Fuzzy Systems for Classification Problems: Mini-Batch Gradient Descent with Uniform Regularization and Batch Normalization
• Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks
• Gradient-based deterministic inversion of geophysical data with Generative Adversarial Networks: is it feasible?
• Intra-Domain Task-Adaptive Transfer Learning to Determine Acute Ischemic Stroke Onset Time
• A global method to identify trees outside of closed-canopy forests with medium-resolution satellite imagery
• Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergence
• Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers
• The DNNLikelihood: enhancing likelihood distribution with Deep Learning
• Large-Scale Generative Data-Free Distillation
• An Adaptive and Momental Bound Method for Stochastic Learning
• AdaX: Adaptive Gradient Descent with Exponential Long Term Memory
• Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates
• On the adequacy of untuned warmup for adaptive optimization
• Audio Captioning using Pre-Trained Large-Scale Language Model Guided by Audio-based Similar Caption Retrieval
• Towards Unified INT8 Training for Convolutional Neural Network
• When Explainability Meets Adversarial Learning: Detecting Adversarial Examples using SHAP Signatures
• Mixing ADAM and SGD: a Combined Optimization Method
• On the Convergence of AdaBound and its Connection to SGD
• 1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed
• LaProp: Separating Momentum and Adaptivity in Adam
• On the Impact of Device and Behavioral Heterogeneity in Federated Learning
• Convergence of Adam for Non-convex Objectives: Relaxed Hyperparameters and Non-ergodic Case
• Positive-Negative Momentum: Manipulating Stochastic Gradient Noise to Improve Generalization
• Spherical View Synthesis for Self-Supervised 360 Depth Estimation
• A Modular Neural Network Based Deep Learning Approach for MIMO Signal Detection
• Deep Learning for 1-Bit Compressed Sensing-based Superimposed CSI Feedback
• A Control Theoretic Framework for Adaptive Gradient Optimizers in Machine Learning
• Characterizing Structural Regularities of Labeled Data in Overparameterized Models
• Convergence Analysis of a Momentum Algorithm with Adaptive Step Size for Non Convex Optimization
• Stochastic quasi-Newton with line-search regularization
• Nested Named Entity Recognition via Second-best Sequence Learning and Decoding
• Training Deep Neural Networks with Adaptive Momentum Inspired by the Quadratic Optimization
• A new regret analysis for Adam-type algorithms
• Projection-Free Adaptive Gradients for Large-Scale Optimization
• On the Convergence of Decentralized Adaptive Gradient Methods
• FCM-RDpA: TSK Fuzzy Regression Model Construction Using Fuzzy C-Means Clustering, Regularization, DropRule, and Powerball AdaBelief
• Convergence Analyses of Online ADAM Algorithm in Convex Setting and Two-Layer ReLU Neural Network
• APMSqueeze: A Communication Efficient Adam-Preconditioned Momentum SGD Algorithm
• On the Distributional Properties of Adaptive Gradients
• The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networks
• Momentum Accelerates the Convergence of Stochastic AUPRC Maximization
• Instance Enhancement Batch Normalization: an Adaptive Regulator of Batch Noise
• FastAdaBelief: Improving Convergence Rate for Belief-based Adaptive Optimizers by Exploiting Strong Convexity
• Parallel and distributed asynchronous adaptive stochastic gradient methods
• MBGD-RDA Training and Rule Pruning for Concise TSK Fuzzy Regression Models
• On the One-sided Convergence of Adam-type Algorithms in Non-convex Non-concave Min-max Optimization
• Variance Reduction on General Adaptive Stochastic Mirror Descent
• On the Evaluation and Real-World Usage Scenarios of Deep Vessel Segmentation for Retinography
• DEAM: Adaptive Momentum with Discriminative Weight for Stochastic Optimization
• On the Last Iterate Convergence of Momentum Methods
• Channel-Directed Gradients for Optimization of Convolutional Neural Networks
• Local Convergence of Adaptive Gradient Descent Optimizers
• Neurons learn slower than they think
• Stochastic Approximate Gradient Descent via the Langevin Algorithm
• Flexible numerical optimization with ensmallen
• MaxVA: Fast Adaptation of Step Sizes by Maximizing Observed Variance of Gradients
• Adaptive Hierarchical Hyper-gradient Descent
• BAMSProd: A Step towards Generalizing the Adaptive Optimization Methods to Deep Binary Model
• An Adaptive Remote Stochastic Gradient Method for Training Neural Networks
• A straightforward line search approach on the expected empirical loss for stochastic deep learning problems
• Toward Communication Efficient Adaptive Gradient Method
• Parabolic Approximation Line Search for DNNs
• A New Adaptive Gradient Method with Gradient Decomposition
• Fit to Measure: Reasoning about Sizes for Robust Object Recognition
• The Number of Steps Needed for Nonconvex Optimization of a Deep Learning Optimizer is a Rational Function of Batch Size