Improving the convergence of SGD through adaptive batch sizes
arXiv:1910.08222
Abstract
Mini-batch stochastic gradient descent (SGD) and variants thereof approximate the objective function's gradient with a small number of training examples, aka the batch size. Small batch sizes require little computation for each model update but can yield high-variance gradient estimates, which poses some challenges for optimization. Conversely, large batches require more computation but can yield higher precision gradient estimates. This work presents a method to adapt the batch size to the model's training loss. For various function classes, we show that our method requires the same order of model updates as gradient descent while requiring the same order of gradient computations as SGD. This method requires evaluating the model's loss on the entire dataset every model update. However, the required computation is greatly reduced by approximating the training loss. We provide experiments that illustrate our methods require fewer model updates without increasing the total amount of computation.
References in corpus (15)
- Adam: A Method for Stochastic Optimization
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- ADADELTA: An Adaptive Learning Rate Method
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
- Don't Decay the Learning Rate, Increase the Batch Size
- Three Factors Influencing Minima in SGD
- AdaGrad stepsizes: Sharp convergence over nonconvex landscapes
- Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolate
- ImageNet pre-trained models with batch normalization
- AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks
- Big Batch SGD: Automated Inference using Adaptive Batch Sizes
- Tight Dimension Independent Lower Bound on the Expected Convergence Rate for Diminishing Step Sizes in SGD
- The YouTube-8M Kaggle Competition: Challenges and Methods
- History-Gradient Aided Batch Size Adaptation for Variance Reduced Algorithms