Acceleration via Fractal Learning Rate Schedules
arXiv:2103.01338
Abstract
In practical applications of iterative first-order optimization, the learning rate schedule remains notoriously difficult to understand and expensive to tune. We demonstrate the presence of these subtleties even in the innocuous case when the objective is a convex quadratic. We reinterpret an iterative algorithm from the numerical analysis literature as what we call the Chebyshev learning rate schedule for accelerating vanilla gradient descent, and show that the problem of mitigating instability leads to a fractal ordering of step sizes. We provide some experiments to challenge conventional beliefs about stable learning rates in deep learning: the fractal schedule enables training to converge with locally unstable updates which make negative progress on the objective.
arXiv commentv2: revisions for ICML 2021
References in corpus (24)
- Adam: A Method for Stochastic Optimization
- Language Models are Few-Shot Learners
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Don't Decay the Learning Rate, Increase the Batch Size
- On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
- A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights
- No More Pesky Learning Rates
- Neural Optimizer Search with Reinforcement Learning
- AdaGrad stepsizes: Sharp convergence over nonconvex landscapes
- Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing
- Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
- Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks
- A geometric alternative to Nesterov's accelerated gradient descent
- An Exponential Learning Rate Schedule for Deep Learning
- Stability and Convergence Trade-off of Iterative Optimization Algorithms
- Faster Neural Network Training with Data Echoing
- Near-optimal method for highly smooth convex optimization
- Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability
- Escaping Saddle Points with Adaptive Gradient Methods
- On Accelerated Methods in Optimization
- Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Rate
- Revisiting the Polyak step size
- Disentangling Adaptive Gradient Methods from Learning Rates
- At Stability's Edge: How to Adjust Hyperparameters to Preserve Minima Selection in Asynchronous Training of Neural Networks?