Empirical study towards understanding line search approximations for training neural networks

Abstract

Choosing appropriate step sizes is critical for reducing the computational cost of training large-scale neural network models. Mini-batch sub-sampling (MBSS) is often employed for computational tractability. However, MBSS introduces a sampling error, that can manifest as a bias or variance in a line search. This is because MBSS can be performed statically, where the mini-batch is updated only when the search direction changes, or dynamically, where the mini-batch is updated every-time the function is evaluated. Static MBSS results in a smooth loss function along a search direction, reflecting low variance but large bias in the estimated "true" (or full batch) minimum. Conversely, dynamic MBSS results in a point-wise discontinuous function, with computable gradients using backpropagation, along a search direction, reflecting high variance but lower bias in the estimated "true" (or full batch) minimum. In this study, quadratic line search approximations are considered to study the quality of function and derivative information to construct approximations for dynamic MBSS loss functions. An empirical study is conducted where function and derivative information are enforced in various ways for the quadratic approximations. The results for various neural network problems show that being selective on what information is enforced helps to reduce the variance of predicted step sizes.

30 pages, 20 figures

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• ADADELTA: An Adaptive Learning Rate Method
• SGDR: Stochastic Gradient Descent with Warm Restarts
• The Marginal Value of Adaptive Gradient Methods in Machine Learning
• Revisiting Small Batch Training for Deep Neural Networks
• No More Pesky Learning Rates
• Not All Samples Are Created Equal: Deep Learning with Importance Sampling
• Adaptive learning rates and parallelization for stochastic, sparse, non-smooth gradients
• VR-SGD: A Simple Stochastic Variance Reduction Method for Machine Learning
• ADASECANT: Robust Adaptive Secant Method for Stochastic Gradient
• Gradient-only line searches: An Alternative to Probabilistic Line Searches

Cited by in corpus (5)

• Investigating the interaction between gradient-only line searches and different activation functions
• GOALS: Gradient-Only Approximations for Line Searches Towards Robust and Consistent Training of Deep Neural Networks
• A straightforward line search approach on the expected empirical loss for stochastic deep learning problems
• Using a one dimensional parabolic model of the full-batch loss to estimate learning rates during training
• Parabolic Approximation Line Search for DNNs