AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients

Abstract

Most popular optimizers for deep learning can be broadly categorized as adaptive methods (e.g. Adam) and accelerated schemes (e.g. stochastic gradient descent (SGD) with momentum). For many models such as convolutional neural networks (CNNs), adaptive methods typically converge faster but generalize worse compared to SGD; for complex settings such as generative adversarial networks (GANs), adaptive methods are typically the default because of their stability.We propose AdaBelief to simultaneously achieve three goals: fast convergence as in adaptive methods, good generalization as in SGD, and training stability. The intuition for AdaBelief is to adapt the stepsize according to the "belief" in the current gradient direction. Viewing the exponential moving average (EMA) of the noisy gradient as the prediction of the gradient at the next time step, if the observed gradient greatly deviates from the prediction, we distrust the current observation and take a small step; if the observed gradient is close to the prediction, we trust it and take a large step. We validate AdaBelief in extensive experiments, showing that it outperforms other methods with fast convergence and high accuracy on image classification and language modeling. Specifically, on ImageNet, AdaBelief achieves comparable accuracy to SGD. Furthermore, in the training of a GAN on Cifar10, AdaBelief demonstrates high stability and improves the quality of generated samples compared to a well-tuned Adam optimizer. Code is available at https://github.com/juntang-zhuang/Adabelief-Optimizer

References in corpus (16)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Decoupled Weight Decay Regularization
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• ADADELTA: An Adaptive Learning Rate Method
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• On the Convergence of Adam and Beyond
• NIPS 2016 Tutorial: Generative Adversarial Networks
• Improving Generalization Performance by Switching from Adam to SGD
• Gradient Descent Maximizes the Margin of Homogeneous Neural Networks
• Quasi-hyperbolic momentum and Adam for deep learning
• Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks
• On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization
• ChainerRL: A Deep Reinforcement Learning Library
• Nostalgic Adam: Weighting more of the past gradients when designing the adaptive learning rate
• AdaX: Adaptive Gradient Descent with Exponential Long Term Memory
• On the distance between two neural networks and the stability of learning

Cited by in corpus (11)

• AdaTask: A Task-aware Adaptive Learning Rate Approach to Multi-task Learning
• Efficient Adaptive Ensembling for Image Classification
• Goal-Aware Neural SAT Solver
• VSMask: Defending Against Voice Synthesis Attack via Real-Time Predictive Perturbation
• Eating Garlic Prevents COVID-19 Infection: Detecting Misinformation on the Arabic Content of Twitter
• Stochastic Anderson Mixing for Nonconvex Stochastic Optimization
• Variance Reduction on General Adaptive Stochastic Mirror Descent
• Digital Signal Processing Techniques for Noise Characterization of Lasers and Optical Frequency Combs: A Tutorial
• Heterogeneous Graph based Deep Learning for Biomedical Network Link Prediction
• Investigating Alternatives to the Root Mean Square for Adaptive Gradient Methods
• Adaptive Gradient Descent Methods for Computing Implied Volatility