A Resizable Mini-batch Gradient Descent based on a Multi-Armed Bandit
arXiv:1711.06424
Abstract
Determining the appropriate batch size for mini-batch gradient descent is always time consuming as it often relies on grid search. This paper considers a resizable mini-batch gradient descent (RMGD) algorithm based on a multi-armed bandit for achieving best performance in grid search by selecting an appropriate batch size at each epoch with a probability defined as a function of its previous success/failure. This probability encourages exploration of different batch size and then later exploitation of batch size with history of success. At each epoch, the RMGD samples a batch size from its probability distribution, then uses the selected batch size for mini-batch gradient descent. After obtaining the validation loss at each epoch, the probability distribution is updated to incorporate the effectiveness of the sampled batch size. The RMGD essentially assists the learning process to explore the possible domain of the batch size and exploit successful batch size. Experimental results show that the RMGD achieves performance better than the best performing single batch size. Furthermore, it, obviously, attains this performance in a shorter amount of time than grid search. It is surprising that the RMGD achieves better performance than grid search.
arXiv comment8 pages, 5 figures, 5 tables
References in corpus (9)
- Striving for Simplicity: The All Convolutional Net
- Don't Decay the Learning Rate, Increase the Batch Size
- On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
- Systematic evaluation of CNN advances on the ImageNet
- Practical recommendations for gradient-based training of deep architectures
- Three Factors Influencing Minima in SGD
- The Effects of Hyperparameters on SGD Training of Neural Networks
- Big Batch SGD: Automated Inference using Adaptive Batch Sizes
- Coupling Adaptive Batch Sizes with Learning Rates