Ranger21: a synergistic deep learning optimizer

Abstract

As optimizers are critical to the performances of neural networks, every year a large number of papers innovating on the subject are published. However, while most of these publications provide incremental improvements to existing algorithms, they tend to be presented as new optimizers rather than composable algorithms. Thus, many worthwhile improvements are rarely seen out of their initial publication. Taking advantage of this untapped potential, we introduce Ranger21, a new optimizer which combines AdamW with eight components, carefully selected after reviewing and testing ideas from the literature. We found that the resulting optimizer provides significantly improved validation accuracy and training speed, smoother training curves, and is even able to train a ResNet50 on ImageNet2012 without Batch Normalization layers. A problem on which AdamW stays systematically stuck in a bad initial state.

arXiv commentfor associated code, see https://github.com/lessw2020/Ranger21

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• On the Variance of the Adaptive Learning Rate and Beyond
• Lookahead Optimizer: k steps forward, 1 step back
• High-Performance Large-Scale Image Recognition Without Normalization
• Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers
• On the adequacy of untuned warmup for adaptive optimization
• Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedule
• Preprint: Norm Loss: An efficient yet effective regularization method for deep neural networks