On the Variance of the Adaptive Learning Rate and Beyond

Abstract

The learning rate warmup heuristic achieves remarkable success in stabilizing training, accelerating convergence and improving generalization for adaptive stochastic optimization algorithms like RMSprop and Adam. Here, we study its mechanism in details. Pursuing the theory behind warmup, we identify a problem of the adaptive learning rate (i.e., it has problematically large variance in the early stage), suggest warmup works as a variance reduction technique, and provide both empirical and theoretical evidence to verify our hypothesis. We further propose RAdam, a new variant of Adam, by introducing a term to rectify the variance of the adaptive learning rate. Extensive experimental results on image classification, language modeling, and neural machine translation verify our intuition and demonstrate the effectiveness and robustness of our proposed method. All implementations are available at: https://github.com/LiyuanLucasLiu/RAdam.

arXiv commentICLR 2020. Fix several typos in the previous version

References in corpus (9)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• ADADELTA: An Adaptive Learning Rate Method
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• On the Convergence of Adam and Beyond
• Training Tips for the Transformer Model
• Adaptive Gradient Methods with Dynamic Bound of Learning Rate
• fairseq: A Fast, Extensible Toolkit for Sequence Modeling
• Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks
• Efficient Contextualized Representation: Language Model Pruning for Sequence Labeling

Cited by in corpus (85)

• Results of the 2020 fastMRI Challenge for Machine Learning MR Image Reconstruction
• Point Transformer
• Iterative Filter Adaptive Network for Single Image Defocus Deblurring
• Review: Deep Learning in Electron Microscopy
• CLIP-Art: Contrastive Pre-training for Fine-Grained Art Classification
• Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
• Motion Puzzle: Arbitrary Motion Style Transfer by Body Part
• Audio Retrieval with Natural Language Queries: A Benchmark Study
• Applications and Techniques for Fast Machine Learning in Science
• StorSeismic: A new paradigm in deep learning for seismic processing
• Lane Detection Model Based on Spatio-Temporal Network With Double Convolutional Gated Recurrent Units
• RR-DnCNN v2.0: Enhanced Restoration-Reconstruction Deep Neural Network for Down-Sampling Based Video Coding
• A Hybrid Spatial-temporal Deep Learning Architecture for Lane Detection
• Towards Fine-grained Human Pose Transfer with Detail Replenishing Network
• Neural Networks Versus Conventional Filters for Inertial-Sensor-based Attitude Estimation
• Reference-based Video Super-Resolution Using Multi-Camera Video Triplets
• Gaze-based dual resolution deep imitation learning for high-precision dexterous robot manipulation
• Short-term daily precipitation forecasting with seasonally-integrated autoencoder
• Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problems
• Getting to 99% Accuracy in Interactive Segmentation
• Vehicle Attribute Recognition by Appearance: Computer Vision Methods for Vehicle Type, Make and Model Classification
• Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors
• AdaTerm: Adaptive T-Distribution Estimated Robust Moments for Noise-Robust Stochastic Gradient Optimization
• Deep Volumetric Ambient Occlusion
• Neutron: An Implementation of the Transformer Translation Model and its Variants
• Exploiting Multiple Sequence Lengths in Fast End to End Training for Image Captioning
• Unconstrained quantitative magnetization transfer imaging: disentangling T1 of the free and semi-solid spin pools
• Rethinking data-driven point spread function modeling with a differentiable optical model
• Only a Matter of Style: Age Transformation Using a Style-Based Regression Model
• Tuning Fairness by Balancing Target Labels
• PLOP: Probabilistic poLynomial Objects trajectory Planning for autonomous driving
• Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedule
• LaProp: Separating Momentum and Adaptivity in Adam
• Using a thousand optimization tasks to learn hyperparameter search strategies
• Detecting mental disorder on social media: a ChatGPT-augmented explainable approach
• Physics-informed machine learning and stray field computation with application to micromagnetic energy minimization
• Convolution-enhanced Evolving Attention Networks
• Online Phase Reconstruction via DNN-based Phase Differences Estimation
• BoolNet: Minimizing The Energy Consumption of Binary Neural Networks
• Adam${}^{+}$: A Stochastic Method with Adaptive Variance Reduction
• How to Learn a Useful Critic? Model-based Action-Gradient-Estimator Policy Optimization
• Multiplicative update rules for accelerating deep learning training and increasing robustness
• Learning spectro-temporal representations of complex sounds with parameterized neural networks
• PTT5: Pretraining and validating the T5 model on Brazilian Portuguese data
• Self-Paced Contrastive Learning for Semi-supervised Medical Image Segmentation with Meta-labels
• Deep learning in magnetic resonance prostate segmentation: A review and a new perspective
• HooliGAN: Robust, High Quality Neural Vocoding
• Dynamic Inference with Neural Interpreters
• Leveraging Lead Bias for Zero-shot Abstractive News Summarization
• Convergence Analysis of a Momentum Algorithm with Adaptive Step Size for Non Convex Optimization
• GNN-XML: Graph Neural Networks for Extreme Multi-label Text Classification
• Revisiting Few-Shot Learning for Facial Expression Recognition
• KISS: Keeping It Simple for Scene Text Recognition
• $F$, $B$, Alpha Matting
• ReFlow6D: Refraction-Guided Transparent Object 6D Pose Estimation via Intermediate Representation Learning
• Solving Bilevel Optimal Bidding Problems Using Deep Convolutional Neural Networks
• Training Neural Networks for and by Interpolation
• Deep Connectomics Networks: Neural Network Architectures Inspired by Neuronal Networks
• Port-Hamiltonian Approach to Neural Network Training
• LQResNet: A Deep Neural Network Architecture for Learning Dynamic Processes
• An Algorithm for Routing Capsules in All Domains
• On the Distributional Properties of Adaptive Gradients
• Ranger21: a synergistic deep learning optimizer
• Simultaneous energy and mass calibration of large-radius jets with the ATLAS detector using a deep neural network
• Large Learning Rate Tames Homogeneity: Convergence and Balancing Effect
• Anomaly Detection By Autoencoder Based On Weighted Frequency Domain Loss
• TreeCaps: Tree-Structured Capsule Networks for Program Source Code Processing
• Match${}^2$: A Matching over Matching Model for Similar Question Identification
• Learning to Correspond Dynamical Systems
• Monocular pose estimation of articulated open surgery tools -- in the wild
• A neural network model that learns differences in diagnosis strategies among radiologists has an improved area under the curve for aneurysm status classification in magnetic resonance angiography image series
• Learn Faster and Forget Slower via Fast and Stable Task Adaptation
• Faster Convergence for Transformer Fine-tuning with Line Search Methods
• Tasting the cake: evaluating self-supervised generalization on out-of-distribution multimodal MRI data
• Enhanced 3D Human Pose Estimation from Videos by using Attention-Based Neural Network with Dilated Convolutions
• On the Trend-corrected Variant of Adaptive Stochastic Optimization Methods
• Hierarchical Attention-based Age Estimation and Bias Estimation
• LogAvgExp Provides a Principled and Performant Global Pooling Operator
• Diffusion tensor driven image registration: a deep learning approach
• Research of Damped Newton Stochastic Gradient Descent Method for Neural Network Training
• Intrinsic Temporal Regularization for High-resolution Human Video Synthesis
• Generative Tomography Reconstruction
• Delta Sampling R-BERT for limited data and low-light action recognition
• Buildings Classification using Very High Resolution Satellite Imagery
• How much progress have we made in neural network training? A New Evaluation Protocol for Benchmarking Optimizers