Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers

Abstract

Choosing the optimizer is considered to be among the most crucial design decisions in deep learning, and it is not an easy one. The growing literature now lists hundreds of optimization methods. In the absence of clear theoretical guidance and conclusive empirical evidence, the decision is often made based on anecdotes. In this work, we aim to replace these anecdotes, if not with a conclusive ranking, then at least with evidence-backed heuristics. To do so, we perform an extensive, standardized benchmark of fifteen particularly popular deep learning optimizers while giving a concise overview of the wide range of possible choices. Analyzing more than $50,000$ individual runs, we contribute the following three points: (i) Optimizer performance varies greatly across tasks. (ii) We observe that evaluating multiple optimizers with default parameters works approximately as well as tuning the hyperparameters of a single, fixed optimizer. (iii) While we cannot discern an optimization method clearly dominating across all tested tasks, we identify a significantly reduced subset of specific optimizers and parameter choices that generally lead to competitive results in our experiments: Adam remains a strong contender, with newer methods failing to significantly and consistently outperform it. Our open-sourced results are available as challenging and well-tuned baselines for more meaningful evaluations of novel optimization methods without requiring any further computational efforts.

arXiv commentRaw results: https://github.com/SirRob1997/Crowded-Valley---Results

References in corpus (77)

• Decoupled Weight Decay Regularization
• ADADELTA: An Adaptive Learning Rate Method
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• On the Convergence of Adam and Beyond
• Large Batch Training of Convolutional Networks
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• Improving Generalization Performance by Switching from Adam to SGD
• Lookahead Optimizer: k steps forward, 1 step back
• No More Pesky Learning Rates
• Neural Optimizer Search with Reinforcement Learning
• Adaptive Gradient Methods with Dynamic Bound of Learning Rate
• On Empirical Comparisons of Optimizers for Deep Learning
• PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization
• Barzilai-Borwein Step Size for Stochastic Gradient Descent
• AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks
• On the Convergence Proof of AMSGrad and a New Version
• Quasi-hyperbolic momentum and Adam for deep learning
• Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks
• AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training
• YellowFin and the Art of Momentum Tuning
• Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis
• Nostalgic Adam: Weighting more of the past gradients when designing the adaptive learning rate
• WNGrad: Learn the Learning Rate in Gradient Descent
• A Walk with SGD
• Shampoo: Preconditioned Stochastic Tensor Optimization
• Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimization
• Practical Gauss-Newton Optimisation for Deep Learning
• An Adaptive and Momental Bound Method for Stochastic Learning
• GADAM: Genetic-Evolutionary ADAM for Deep Neural Network Optimization
• Online Adaptive Methods, Universality and Acceleration
• AdaX: Adaptive Gradient Descent with Exponential Long Term Memory
• PAGE: A Simple and Optimal Probabilistic Gradient Estimator for Nonconvex Optimization
• Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates
• AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks
• AdaScale SGD: A User-Friendly Algorithm for Distributed Training
• Normalized Direction-preserving Adam
• ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks
• Does Adam optimizer keep close to the optimal point?
• Memory-Efficient Adaptive Optimization
• Mixing ADAM and SGD: a Combined Optimization Method
• Adaptive learning rates and parallelization for stochastic, sparse, non-smooth gradients
• CoolMomentum: A Method for Stochastic Optimization by Langevin Dynamics with Simulated Annealing
• Biased Stochastic First-Order Methods for Conditional Stochastic Optimization and Applications in Meta Learning
• Eve: A Gradient Based Optimization Method with Locally and Globally Adaptive Learning Rates
• The Role of Memory in Stochastic Optimization
• Gradient-only line searches: An Alternative to Probabilistic Line Searches
• On the Generalization of Stochastic Gradient Descent with Momentum
• Compositional ADAM: An Adaptive Compositional Solver
• Second-order step-size tuning of SGD for non-convex optimization
• AdaS: Adaptive Scheduling of Stochastic Gradients
• Stochastic Gradient Descent with Nonlinear Conjugate Gradient-Style Adaptive Momentum
• TAdam: A Robust Stochastic Gradient Optimizer
• Adaloss: Adaptive Loss Function for Landmark Localization
• Momentum-based variance-reduced proximal stochastic gradient method for composite nonconvex stochastic optimization
• Training Neural Networks for and by Interpolation
• Learning by Turning: Neural Architecture Aware Optimisation
• AutoLRS: Automatic Learning-Rate Schedule by Bayesian Optimization on the Fly
• S-SGD: Symmetrical Stochastic Gradient Descent with Weight Noise Injection for Reaching Flat Minima
• AdaSGD: Bridging the gap between SGD and Adam
• Momentum with Variance Reduction for Nonconvex Composition Optimization
• A Generalizable Approach to Learning Optimizers
• Expectigrad: Fast Stochastic Optimization with Robust Convergence Properties
• Stochastic Runge-Kutta methods and adaptive SGD-G2 stochastic gradient descent
• Stochastic Gradient Methods with Block Diagonal Matrix Adaptation
• Gravilon: Applications of a New Gradient Descent Method to Machine Learning
• On the Trend-corrected Variant of Adaptive Stochastic Optimization Methods
• A New Accelerated Stochastic Gradient Method with Momentum
• Gravity Optimizer: a Kinematic Approach on Optimization in Deep Learning
• Adaptive Gradient Methods Can Be Provably Faster than SGD after Finite Epochs
• On Higher-order Moments in Adam
• CProp: Adaptive Learning Rate Scaling from Past Gradient Conformity
• A Variant of Gradient Descent Algorithm Based on Gradient Averaging
• ICLR Reproducibility Challenge Report (Padam : Closing The Generalization Gap Of Adaptive Gradient Methods in Training Deep Neural Networks)
• A Probabilistically Motivated Learning Rate Adaptation for Stochastic Optimization
• Learning in Deep Neural Networks Using a Biologically Inspired Optimizer
• GOALS: Gradient-Only Approximations for Line Searches Towards Robust and Consistent Training of Deep Neural Networks
• Second-order Information in First-order Optimization Methods

Cited by in corpus (16)

• Review: Deep Learning in Electron Microscopy
• Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimization
• Are we Forgetting about Compositional Optimisers in Bayesian Optimisation?
• A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes
• Stochastic Training is Not Necessary for Generalization
• FCM-RDpA: TSK Fuzzy Regression Model Construction Using Fuzzy C-Means Clustering, Regularization, DropRule, and Powerball AdaBelief
• Ranger21: a synergistic deep learning optimizer
• Deep Neural Network Training with Frank-Wolfe
• Deep learning for the modeling and inverse design of radiative heat transfer
• Meta-Learning Bidirectional Update Rules
• Explainability-aided Domain Generalization for Image Classification
• Length Scale Control in Topology Optimization using Fourier Enhanced Neural Networks
• Training Aware Sigmoidal Optimizer
• Surrogate Model Based Hyperparameter Tuning for Deep Learning with SPOT
• A straightforward line search approach on the expected empirical loss for stochastic deep learning problems
• The Number of Steps Needed for Nonconvex Optimization of a Deep Learning Optimizer is a Rational Function of Batch Size