WNGrad: Learn the Learning Rate in Gradient Descent

Abstract

Adjusting the learning rate schedule in stochastic gradient methods is an important unresolved problem which requires tuning in practice. If certain parameters of the loss function such as smoothness or strong convexity constants are known, theoretical learning rate schedules can be applied. However, in practice, such parameters are not known, and the loss function of interest is not convex in any case. The recently proposed batch normalization reparametrization is widely adopted in most neural network architectures today because, among other advantages, it is robust to the choice of Lipschitz constant of the gradient in loss function, allowing one to set a large learning rate without worry. Inspired by batch normalization, we propose a general nonlinear update rule for the learning rate in batch and stochastic gradient descent so that the learning rate can be initialized at a high value, and is subsequently decreased according to gradient observations along the way. The proposed method is shown to achieve robustness to the relationship between the learning rate and the Lipschitz constant, and near-optimal convergence rates in both the batch and stochastic settings ($O(1/T)$ for smooth loss in the batch setting, and $O(1/\sqrt{T})$ for convex loss in the stochastic setting). We also show through numerical evidence that such robustness of the proposed method extends to highly nonconvex and possibly non-smooth loss function in deep learning problems.Our analysis establishes some first theoretical understanding into the observed robustness for batch normalization and weight normalization.

arXiv comment10 pages, 3 figures, conference

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• ADADELTA: An Adaptive Learning Rate Method
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• The Marginal Value of Adaptive Gradient Methods in Machine Learning
• Stochastic Gradient Descent for Non-smooth Optimization: Convergence Results and Optimal Averaging Schemes
• AdaGrad stepsizes: Sharp convergence over nonconvex landscapes
• AdaBatch: Efficient Gradient Aggregation Rules for Sequential and Parallel Stochastic Gradient Methods

Cited by in corpus (23)

• Theoretical Analysis of Auto Rate-Tuning by Batch Normalization
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• An Exponential Learning Rate Schedule for Deep Learning
• Global Convergence of Adaptive Gradient Methods for An Over-parameterized Neural Network
• Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers
• Deep Frank-Wolfe For Neural Network Optimization
• On the Heavy-Tailed Theory of Stochastic Gradient Descent for Deep Neural Networks
• Bayesian Inference for Large Scale Image Classification
• Optimization Theory for ReLU Neural Networks Trained with Normalization Layers
• Gradient-only line searches: An Alternative to Probabilistic Line Searches
• Stochastic Gradient Descent with Polyak's Learning Rate
• Convergence Analysis of a Momentum Algorithm with Adaptive Step Size for Non Convex Optimization
• Linear Convergence of Adaptive Stochastic Gradient Descent
• Training Neural Networks for and by Interpolation
• LOSSGRAD: automatic learning rate in gradient descent
• Implicit Regularization and Convergence for Weight Normalization
• Robust Implicit Regularization via Weight Normalization
• A Stochastic Objective-Function-Free Adaptive Regularization Method with Optimal Complexity
• Gradient-only line searches to automatically determine learning rates for a variety of stochastic training algorithms
• AutoDrop: Training Deep Learning Models with Automatic Learning Rate Drop
• Feedback Control for Online Training of Neural Networks
• Flexible numerical optimization with ensmallen
• Data-driven Algorithm Selection and Parameter Tuning: Two Case studies in Optimization and Signal Processing