On the adequacy of untuned warmup for adaptive optimization

Abstract

Adaptive optimization algorithms such as Adam are widely used in deep learning. The stability of such algorithms is often improved with a warmup schedule for the learning rate. Motivated by the difficulty of choosing and tuning warmup schedules, recent work proposes automatic variance rectification of Adam's adaptive learning rate, claiming that this rectified approach ("RAdam") surpasses the vanilla Adam algorithm and reduces the need for expensive tuning of Adam with warmup. In this work, we refute this analysis and provide an alternative explanation for the necessity of warmup based on the magnitude of the update term, which is of greater relevance to training stability. We then provide some "rule-of-thumb" warmup schedules, and we demonstrate that simple untuned warmup of Adam performs more-or-less identically to RAdam in typical practical settings. We conclude by suggesting that practitioners stick to linear warmup with Adam, with a sensible default being linear warmup over $2/(1-{\beta }_2)$ training iterations.

arXiv commentAAAI 2021

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Decoupled Weight Decay Regularization
• Convolutional Sequence to Sequence Learning
• Don't Decay the Learning Rate, Increase the Batch Size
• Improving Generalization Performance by Switching from Adam to SGD
• Use What You Have: Video Retrieval Using Representations From Collaborative Experts
• Adaptive Gradient Methods with Dynamic Bound of Learning Rate
• Quasi-hyperbolic momentum and Adam for deep learning
• Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks
• Online Learning Rate Adaptation with Hypergradient Descent
• YellowFin and the Art of Momentum Tuning
• A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillation

Cited by in corpus (8)

• The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
• Vilio: State-of-the-art Visio-Linguistic Models applied to Hateful Memes
• Are we pretraining it right? Digging deeper into visio-linguistic pretraining
• Reflection-Equivariant Diffusion for 3D Structure Determination from Isotopologue Rotational Spectra in Natural Abundance
• On the Distributional Properties of Adaptive Gradients
• Ranger21: a synergistic deep learning optimizer
• Is High Variance Unavoidable in RL? A Case Study in Continuous Control
• MaxVA: Fast Adaptation of Step Sizes by Maximizing Observed Variance of Gradients