1 citations · 3 across the 5 of their papers we have counts for
7 papers · 1 filter
GOALS: Gradient-Only Approximations for Line Searches Towards Robust and Consistent Training of Deep Neural Networks
Younghwan Chae, Daniel N. Wilke, Dominic Kafka
Mini-batch sub-sampling (MBSS) is favored in deep neural network training to reduce the computational cost. Still, it introduces an inherent sampling error, making the selection of…
Gradient-only line searches to automatically determine learning rates for a variety of stochastic training algorithms
Dominic Kafka, Daniel Nicolas Wilke
Gradient-only and probabilistic line searches have recently reintroduced the ability to adaptively determine learning rates in dynamic mini-batch sub-sampled neural network trainin…
Investigating the interaction between gradient-only line searches and different activation functions
D. Kafka, Daniel. N. Wilke
Gradient-only line searches (GOLS) adaptively determine step sizes along search directions for discontinuous loss functions resulting from dynamic mini-batch sub-sampling in neural…
Resolving learning rates adaptively by locating Stochastic Non-Negative Associated Gradient Projection Points using line searches
Dominic Kafka, Daniel N. Wilke
Learning rates in stochastic neural network training are currently determined a priori to training, using expensive manual or automated iterative tuning. This study proposes gradie…
Empirical study towards understanding line search approximations for training neural networks
Younghwan Chae, Daniel N. Wilke
Choosing appropriate step sizes is critical for reducing the computational cost of training large-scale neural network models. Mini-batch sub-sampling (MBSS) is often employed for…
Gradient-only line searches: An Alternative to Probabilistic Line Searches
Dominic Kafka, Daniel Wilke
Step sizes in neural network training are largely determined using predetermined rules such as fixed learning rates and learning rate schedules. These require user input or expensi…