Gradient-only line searches: An Alternative to Probabilistic Line Searches

Abstract

Step sizes in neural network training are largely determined using predetermined rules such as fixed learning rates and learning rate schedules. These require user input or expensive global optimization strategies to determine their functional form and associated hyperparameters. Line searches are capable of adaptively resolving learning rate schedules. However, due to discontinuities induced by mini-batch sub-sampling, they have largely fallen out of favour. Notwithstanding, probabilistic line searches, which use statistical surrogates over a limited spatial domain, have recently demonstrated viability in resolving learning rates for stochastic loss functions. This paper introduces an alternative paradigm, Gradient-Only Line Searches that are Inexact (GOLS-I), as an alternative strategy to automatically determine learning rates in stochastic loss functions over a range of 15 orders of magnitude without the use of surrogates. We show that GOLS-I is a competitive strategy to reliably determine step sizes, adding high value in terms of performance, while being easy to implement.

25 Pages, 12 Figures, to be submitted to a journal

References in corpus (10)

• SGDR: Stochastic Gradient Descent with Warm Restarts
• Optimal Distributed Online Prediction using Mini-Batches
• Qualitatively characterizing neural network optimization problems
• WNGrad: Learn the Learning Rate in Gradient Descent
• A Stochastic Line Search Method with Convergence Rate Analysis
• A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks
• Stochastic quasi-Newton with line-search regularization
• Algorithms for solving optimization problems arising from deep neural net models: smooth problems
• Stochastic quasi-Newton with adaptive step lengths for large-scale problems
• Traversing the noise of dynamic mini-batch sub-sampled loss functions: A visual guide

Cited by in corpus (8)

• Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers
• Empirical study towards understanding line search approximations for training neural networks
• GOALS: Gradient-Only Approximations for Line Searches Towards Robust and Consistent Training of Deep Neural Networks
• Traversing the noise of dynamic mini-batch sub-sampled loss functions: A visual guide
• Gradient-only line searches to automatically determine learning rates for a variety of stochastic training algorithms
• Using a one dimensional parabolic model of the full-batch loss to estimate learning rates during training
• Parabolic Approximation Line Search for DNNs
• A straightforward line search approach on the expected empirical loss for stochastic deep learning problems