Coupling Adaptive Batch Sizes with Learning Rates

Abstract

Mini-batch stochastic gradient descent and variants thereof have become standard for large-scale empirical risk minimization like the training of neural networks. These methods are usually used with a constant batch size chosen by simple empirical inspection. The batch size significantly influences the behavior of the stochastic optimization algorithm, though, since it determines the variance of the gradient estimates. This variance also changes over the optimization process; when using a constant batch size, stability and convergence is thus often enforced by means of a (manually tuned) decreasing learning rate schedule. We propose a practical method for dynamic batch size adaptation. It estimates the variance of the stochastic gradients and adapts the batch size to decrease the variance proportionally to the value of the objective function, removing the need for the aforementioned learning rate decrease. In contrast to recent related work, our algorithm couples the batch size to the learning rate, directly reflecting the known relationship between the two. On popular image classification benchmarks, our batch size adaptation yields faster optimization convergence, while simultaneously simplifying learning rate tuning. A TensorFlow implementation is available.

Thirty-Third Conference on Uncertainty in Artificial Intelligence (UAI), 2017, (accepted)

References in corpus (3)

• ADADELTA: An Adaptive Learning Rate Method
• Stop Wasting My Gradients: Practical SVRG
• Importance Sampling for Minibatches

Cited by in corpus (13)

• Don't Decay the Learning Rate, Increase the Batch Size
• An Empirical Model of Large-Batch Training
• A Progressive Batching L-BFGS Method for Machine Learning
• Large batch size training of neural networks with adversarial training and second-order information
• Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGD
• Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning
• Estimation of discrete choice models with hybrid stochastic adaptive batch size algorithms
• Large Scale Private Learning via Low-rank Reparametrization
• Robust Learning Rate Selection for Stochastic Optimization via Splitting Diagnostic
• Variance reduction for Riemannian non-convex optimization with batch size adaptation
• High Throughput Synchronous Distributed Stochastic Gradient Descent
• Continuous-time Models for Stochastic Optimization Algorithms
• A Resizable Mini-batch Gradient Descent based on a Multi-Armed Bandit