A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes

Abstract

Recently the LARS and LAMB optimizers have been proposed for training neural networks faster using large batch sizes. LARS and LAMB add layer-wise normalization to the update rules of Heavy-ball momentum and Adam, respectively, and have become popular in prominent benchmarks and deep learning libraries. However, without fair comparisons to standard optimizers, it remains an open question whether LARS and LAMB have any benefit over traditional, generic algorithms. In this work we demonstrate that standard optimization algorithms such as Nesterov momentum and Adam can match or exceed the results of LARS and LAMB at large batch sizes. Our results establish new, stronger baselines for future comparisons at these batch sizes and shed light on the difficulties of comparing optimizers for neural network training more generally.

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Large Batch Training of Convolutional Networks
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• Benchmarking TPU, GPU, and CPU Platforms for Deep Learning
• On Empirical Comparisons of Optimizers for Deep Learning
• MLPerf Training Benchmark
• Image Classification at Supercomputer Scale
• Understanding Short-Horizon Bias in Stochastic Meta-Optimization
• Scale MLPerf-0.6 models on Google TPU-v3 Pods
• Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers
• Disentangling Adaptive Gradient Methods from Learning Rates

Cited by in corpus (4)

• Scalable Second Order Optimization for Deep Learning
• Large-Scale Deep Learning Optimizations: A Comprehensive Survey
• Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks
• Training Efficiency and Robustness in Deep Learning