On Empirical Comparisons of Optimizers for Deep Learning

Abstract

Selecting an optimizer is a central step in the contemporary deep learning pipeline. In this paper, we demonstrate the sensitivity of optimizer comparisons to the hyperparameter tuning protocol. Our findings suggest that the hyperparameter search space may be the single most important factor explaining the rankings obtained by recent empirical comparisons in the literature. In fact, we show that these results can be contradicted when hyperparameter search spaces are changed. As tuning effort grows without bound, more general optimizers should never underperform the ones they can approximate (i.e., Adam should never perform worse than momentum), but recent attempts to compare optimizers either assume these inclusion relationships are not practically relevant or restrict the hyperparameters in ways that break the inclusions. In our experiments, we find that inclusion relationships between optimizers matter in practice and always predict optimizer comparisons. In particular, we find that the popular adaptive gradient methods never underperform momentum or gradient descent. We also report practical tips around tuning often ignored hyperparameters of adaptive gradient methods and raise concerns about fairly benchmarking optimizers for neural network training.

References in corpus (19)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Decoupled Weight Decay Regularization
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Practical Bayesian Optimization of Machine Learning Algorithms
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Striving for Simplicity: The All Convolutional Net
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• On the Convergence of Adam and Beyond
• A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay
• AutoAugment: Learning Augmentation Policies from Data
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• Improving Generalization Performance by Switching from Adam to SGD
• On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization
• Quasi-hyperbolic momentum and Adam for deep learning
• Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov acceleration
• Critical Hyper-Parameters: No Random, No Cry
• AdaShift: Decorrelation and Convergence of Adaptive Learning Rate Methods
• DeepOBS: A Deep Learning Optimizer Benchmark Suite

Cited by in corpus (43)

• Learning Transferable Visual Models From Natural Language Supervision
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• Sustainable AI: Environmental Implications, Challenges and Opportunities
• Review: Deep Learning in Electron Microscopy
• Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift
• AI Security for Geoscience and Remote Sensing: Challenges and Future Trends
• Bump Hunting in Latent Space
• The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learning
• Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers
• Adaptive Gradient Quantization for Data-Parallel SGD
• An Empirical Comparison of Optimizers for Quantum Machine Learning with SPSA-based Gradients
• Near-optimal control of dynamical systems with neural ordinary differential equations
• A Shallow Ritz Method for Elliptic Problems with Singular Sources
• Optimizer Benchmarking Needs to Account for Hyperparameter Tuning
• Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves
• Using a thousand optimization tasks to learn hyperparameter search strategies
• Visualizing high-dimensional loss landscapes with Hessian directions
• Training Process of Unsupervised Learning Architecture for Gravity Spy Dataset
• Large-Scale Deep Learning Optimizations: A Comprehensive Survey
• Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Training
• Disentangling Adaptive Gradient Methods from Learning Rates
• A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes
• Is SGD a Bayesian sampler? Well, almost
• Beyond Accuracy: An Empirical Study on Unit Testing in Open-source Deep Learning Projects
• A Loss Curvature Perspective on Training Instability in Deep Learning
• AdaSGD: Bridging the gap between SGD and Adam
• Gradient descent with momentum --- to accelerate or to super-accelerate?
• A Generalizable Approach to Learning Optimizers
• On the Distributional Properties of Adaptive Gradients
• Dual Averaging is Surprisingly Effective for Deep Learning Optimization
• Individual predictions matter: Assessing the effect of data ordering in training fine-tuned CNNs for medical imaging
• Reverse engineering learned optimizers reveals known and novel mechanisms
• Gravity Optimizer: a Kinematic Approach on Optimization in Deep Learning
• Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks
• L2M: Practical posterior Laplace approximation with optimization-driven second moment estimation
• Self-Tuning Stochastic Optimization with Curvature-Aware Gradient Filtering
• Training Efficiency and Robustness in Deep Learning
• Revisiting the Role of Euler Numerical Integration on Acceleration and Stability in Convex Optimization
• Tensor Normal Training for Deep Learning Models
• Don't Sweep your Learning Rate under the Rug: A Closer Look at Cross-modal Transfer of Pretrained Transformers
• Complex Momentum for Optimization in Games
• How much progress have we made in neural network training? A New Evaluation Protocol for Benchmarking Optimizers
• Stable Anderson Acceleration for Deep Learning