Online Learning Rate Adaptation with Hypergradient Descent

Abstract

We introduce a general method for improving the convergence rate of gradient-based optimizers that is easy to implement and works well in practice. We demonstrate the effectiveness of the method in a range of optimization problems by applying it to stochastic gradient descent, stochastic gradient descent with Nesterov momentum, and Adam, showing that it significantly reduces the need for the manual tuning of the initial learning rate for these commonly used algorithms. Our method works by dynamically updating the learning rate during optimization using the gradient with respect to the learning rate of the update rule itself. Computing this "hypergradient" needs little additional computation, requires only one extra copy of the original gradient to be stored in memory, and relies upon nothing more than what is provided by reverse-mode automatic differentiation.

11 pages, 4 figures

References in corpus (5)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Practical Bayesian Optimization of Machine Learning Algorithms
• Gradient-based Hyperparameter Optimization through Reversible Learning
• No More Pesky Learning Rates
• Practical recommendations for gradient-based training of deep architectures

Cited by in corpus (25)

• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• Learning an Adaptive Learning Rate Schedule
• learn2learn: A Library for Meta-Learning Research
• Adaptive Scheduling for Multi-Task Learning
• Robust MAML: Prioritization task buffer with adaptive learning process for model-agnostic meta-learning
• On the adequacy of untuned warmup for adaptive optimization
• AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks
• Gradient Descent: The Ultimate Optimizer
• Improved Bilevel Model: Fast and Optimal Algorithm with Theoretical Guarantee
• AdaS: Adaptive Scheduling of Stochastic Gradients
• Joint Search of Data Augmentation Policies and Network Architectures
• On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment
• A Generalizable Approach to Learning Optimizers
• Doubly Adaptive Scaled Algorithm for Machine Learning Using Second-Order Information
• Block stochastic gradient descent for large-scale tomographic reconstruction in a parallel network
• Population-Based Evolution Optimizes a Meta-Learning Objective
• Self-Tuning Stochastic Optimization with Curvature-Aware Gradient Filtering
• Parabolic Approximation Line Search for DNNs
• A straightforward line search approach on the expected empirical loss for stochastic deep learning problems
• Generalized Reinforcement Meta Learning for Few-Shot Optimization
• Meta-Regularization: An Approach to Adaptive Choice of the Learning Rate in Gradient Descent
• Ada-Segment: Automated Multi-loss Adaptation for Panoptic Segmentation
• Learning by Examples Based on Multi-level Optimization
• Adaptive Hierarchical Hyper-gradient Descent
• Is the Meta-Learning Idea Able to Improve the Generalization of Deep Neural Networks on the Standard Supervised Learning?