AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights

Abstract

Normalization techniques are a boon for modern deep learning. They let weights converge more quickly with often better generalization performances. It has been argued that the normalization-induced scale invariance among the weights provides an advantageous ground for gradient descent (GD) optimizers: the effective step sizes are automatically reduced over time, stabilizing the overall training procedure. It is often overlooked, however, that the additional introduction of momentum in GD optimizers results in a far more rapid reduction in effective step sizes for scale-invariant weights, a phenomenon that has not yet been studied and may have caused unwanted side effects in the current practice. This is a crucial issue because arguably the vast majority of modern deep neural networks consist of (1) momentum-based GD (e.g. SGD or Adam) and (2) scale-invariant parameters. In this paper, we verify that the widely-adopted combination of the two ingredients lead to the premature decay of effective step sizes and sub-optimal model performances. We propose a simple and effective remedy, SGDP and AdamP: get rid of the radial component, or the norm-increasing direction, at each optimizer step. Because of the scale invariance, this modification only alters the effective step sizes without changing the effective update directions, thus enjoying the original convergence properties of GD optimizers. Given the ubiquity of momentum GD and scale invariance in machine learning, we have evaluated our methods against the baselines on 13 benchmarks. They range from vision tasks like classification (e.g. ImageNet), retrieval (e.g. CUB and SOP), and detection (e.g. COCO) to language modelling (e.g. WikiText) and audio classification (e.g. DCASE) tasks. We verify that our solution brings about uniform gains in those benchmarks. Source code is available at https://github.com/clovaai/AdamP.

Accepted at ICLR 2021. First two authors contributed equally

References in corpus (22)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Decoupled Weight Decay Regularization
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• SGDR: Stochastic Gradient Descent with Warm Restarts
• On the Convergence of Adam and Beyond
• Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
• AutoAugment: Learning Augmentation Policies from Data
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• Large Batch Training of Convolutional Networks
• Pointer Sentinel Mixture Models
• Lookahead Optimizer: k steps forward, 1 step back
• Natural Adversarial Examples
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
• NSML: Meet the MLaaS platform with a real-world case study
• Norm matters: efficient and accurate normalization schemes in deep networks
• Learning De-biased Representations with Biased Representations
• Theoretical Analysis of Auto Rate-Tuning by Batch Normalization
• An Empirical Evaluation on Robustness and Uncertainty of Regularization Methods
• Rethinking Channel Dimensions for Efficient Model Design
• Optimization Theory for ReLU Neural Networks Trained with Normalization Layers

Cited by in corpus (7)

• ResNet strikes back: An improved training procedure in timm
• Deep Context- and Relation-Aware Learning for Aspect-based Sentiment Analysis
• Rethinking Channel Dimensions for Efficient Model Design
• The Training Process of Many Deep Networks Explores the Same Low-Dimensional Manifold
• Multi-Attention Based Ultra Lightweight Image Super-Resolution
• On the Relevance of Temporal Features for Medical Ultrasound Video Recognition
• Few-shot Compositional Font Generation with Dual Memory