Disentangling Adaptive Gradient Methods from Learning Rates

Abstract

We investigate several confounding factors in the evaluation of optimization algorithms for deep learning. Primarily, we take a deeper look at how adaptive gradient methods interact with the learning rate schedule, a notoriously difficult-to-tune hyperparameter which has dramatic effects on the convergence and generalization of neural network training. We introduce a "grafting" experiment which decouples an update's magnitude from its direction, finding that many existing beliefs in the literature may have arisen from insufficient isolation of the implicit schedule of step sizes. Alongside this contribution, we present some empirical and theoretical retrospectives on the generalization of adaptive gradient methods, aimed at bringing more clarity to this space.

References in corpus (28)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• ADADELTA: An Adaptive Learning Rate Method
• On the difficulty of training Recurrent Neural Networks
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• SGDR: Stochastic Gradient Descent with Warm Restarts
• On the Convergence of Adam and Beyond
• Reconciling modern machine learning practice and the bias-variance trade-off
• One weird trick for parallelizing convolutional neural networks
• Large Batch Training of Convolutional Networks
• Improving Generalization Performance by Switching from Adam to SGD
• Deep Reinforcement Learning that Matters
• DyNet: The Dynamic Neural Network Toolkit
• Training Tips for the Transformer Model
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• The generalization error of random features regression: Precise asymptotics and double descent curve
• On Empirical Comparisons of Optimizers for Deep Learning
• Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
• Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
• Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks
• Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks
• A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillation
• An Exponential Learning Rate Schedule for Deep Learning
• Memory-Efficient Adaptive Optimization
• Efficient Full-Matrix Adaptive Regularization
• Span-Based Constituency Parsing with a Structure-Label System and Provably Optimal Dynamic Oracles

Cited by in corpus (6)

• Scalable Second Order Optimization for Deep Learning
• Shape Matters: Understanding the Implicit Bias of the Noise Covariance
• How to decay your learning rate
• A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes
• Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes
• Acceleration via Fractal Learning Rate Schedules