No More Pesky Learning Rates

Abstract

The performance of stochastic gradient descent (SGD) depends critically on how learning rates are tuned and decreased over time. We propose a method to automatically adjust multiple learning rates so as to minimize the expected error at any one time. The method relies on local gradient variations across samples. In our approach, learning rates can increase as well as decrease, making it suitable for non-stationary problems. Using a number of convex and non-convex learning tasks, we show that the resulting algorithm matches the performance of SGD or other adaptive approaches with their best settings obtained through systematic search, and effectively removes the need for learning rate tuning.

References in corpus (2)

• Towards Optimal One Pass Large Scale Learning with Averaged Stochastic Gradient Descent
• Estimating the Hessian by Back-propagating Curvature

Cited by in corpus (96)

• Deep Learning in Neural Networks: An Overview
• Prioritized Experience Replay
• Unsupervised Deep Feature Extraction for Remote Sensing Image Classification
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• Lookahead Optimizer: k steps forward, 1 step back
• Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks
• Recent Advances in Convolutional Neural Networks
• Neural Optimizer Search with Reinforcement Learning
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• An Empirical Model of Large-Batch Training
• Optimization for deep learning: theory and algorithms
• Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples
• RNADE: The real-valued neural autoregressive density-estimator
• Learning values across many orders of magnitude
• Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradients
• Implicit Regularization in Deep Learning
• Online Learning Rate Adaptation with Hypergradient Descent
• Reducing Noise in GAN Training with Variance Reduced Extragradient
• Strong error analysis for stochastic gradient descent optimization algorithms
• Unit Tests for Stochastic Optimization
• Convolutional Sparse Kernel Network for Unsupervised Medical Image Analysis
• Understanding Short-Horizon Bias in Stochastic Meta-Optimization
• Learning an Adaptive Learning Rate Schedule
• Big Batch SGD: Automated Inference using Adaptive Batch Sizes
• Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers
• Large batch size training of neural networks with adversarial training and second-order information
• Stochastic Dual Coordinate Ascent with Adaptive Probabilities
• Implementation of Fruits Recognition Classifier using Convolutional Neural Network Algorithm for Observation of Accuracies for Various Hidden Layers
• Stochastic Optimization for Performative Prediction
• Probabilistic Line Searches for Stochastic Optimization
• Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates
• Deep Frank-Wolfe For Neural Network Optimization
• Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability
• On the distance between two neural networks and the stability of learning
• Reinforcement Learning for Learning Rate Control
• Correlation Maximized Structural Similarity Loss for Semantic Segmentation
• Stochastic Optimization for Deep CCA via Nonlinear Orthogonal Iterations
• Demystifying Learning Rate Policies for High Accuracy Training of Deep Neural Networks
• Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model
• Gradient Amplification: An efficient way to train deep neural networks
• ADASECANT: Robust Adaptive Secant Method for Stochastic Gradient
• An Empirical Study of Large-Batch Stochastic Gradient Descent with Structured Covariance Noise
• AdaS: Adaptive Scheduling of Stochastic Gradients
• Stochastic Gradient Descent: Going As Fast As Possible But Not Faster
• Dynamic of Stochastic Gradient Descent with State-Dependent Noise
• Empirical study towards understanding line search approximations for training neural networks
• Training Deep Networks without Learning Rates Through Coin Betting
• Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes
• The Bayesian Learning Rule
• DeepOBS: A Deep Learning Optimizer Benchmark Suite
• Training Neural Networks for and by Interpolation
• On Accelerating Distributed Convex Optimizations
• Learning Strict Identity Mappings in Deep Residual Networks
• No more meta-parameter tuning in unsupervised sparse feature learning
• Block stochastic gradient iteration for convex and nonconvex optimization
• Gradient descent revisited via an adaptive online learning rate
• AutoLRS: Automatic Learning-Rate Schedule by Bayesian Optimization on the Fly
• A Simple Dynamic Learning Rate Tuning Algorithm For Automated Training of DNNs
• Speed learning on the fly
• SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticality
• Towards White-box Benchmarks for Algorithm Control
• The Nonlinearity Coefficient - A Practical Guide to Neural Architecture Design
• Data-Dependent Path Normalization in Neural Networks
• Scalable Probabilistic Entity-Topic Modeling
• Acceleration via Fractal Learning Rate Schedules
• Accelerating Mini-batch SARAH by Step Size Rules
• Merging Deterministic Policy Gradient Estimations with Varied Bias-Variance Tradeoff for Effective Deep Reinforcement Learning
• Coordinate Descent with Online Adaptation of Coordinate Frequencies
• TDprop: Does Jacobi Preconditioning Help Temporal Difference Learning?
• Reverse engineering learned optimizers reveals known and novel mechanisms
• Microscopic Advances with Large-Scale Learning: Stochastic Optimization for Cryo-EM
• AutoDrop: Training Deep Learning Models with Automatic Learning Rate Drop
• Particle Filtering Methods for Stochastic Optimization with Application to Large-Scale Empirical Risk Minimization
• Data Sampling Strategies in Stochastic Algorithms for Empirical Risk Minimization
• Patient Flow Prediction via Discriminative Learning of Mutually-Correcting Processes
• Variational Inference for Shrinkage Priors: The R package vir
• A Probabilistically Motivated Learning Rate Adaptation for Stochastic Optimization
• Blind Descent: A Prequel to Gradient Descent
• CProp: Adaptive Learning Rate Scaling from Past Gradient Conformity
• Risk-Averse Approximate Dynamic Programming with Quantile-Based Risk Measures
• Discontinuous Piecewise Polynomial Neural Networks
• An Adaptive Online HDP-HMM for Segmentation and Classification of Sequential Data
• On Generalization of Adaptive Methods for Over-parameterized Linear Regression
• The duality structure gradient descent algorithm: analysis and applications to neural networks
• Convergence Analysis of Gradient Descent Algorithms with Proportional Updates
• MaxVA: Fast Adaptation of Step Sizes by Maximizing Observed Variance of Gradients
• Automatic and Simultaneous Adjustment of Learning Rate and Momentum for Stochastic Gradient Descent
• Post-Selections in AI and How to Avoid Them
• LRTuner: A Learning Rate Tuner for Deep Neural Networks
• POLA: Online Time Series Prediction by Adaptive Learning Rates
• A New Training Method for Feedforward Neural Networks Based on Geometric Contraction Property of Activation Functions
• Cost-Sensitive Approach to Batch Size Adaptation for Gradient Descent
• Linear Speedup in Personalized Collaborative Learning
• Data-driven Algorithm Selection and Parameter Tuning: Two Case studies in Optimization and Signal Processing
• Reparametrizing gradient descent
• Optimizing Convergence for Iterative Learning of ARIMA for Stationary Time Series