Inefficiency of K-FAC for Large Batch Size Training
arXiv:1903.06237
Abstract
In stochastic optimization, using large batch sizes during training can leverage parallel resources to produce faster wall-clock training times per training epoch. However, for both training loss and testing error, recent results analyzing large batch Stochastic Gradient Descent (SGD) have found sharp diminishing returns, beyond a certain critical batch size. In the hopes of addressing this, it has been suggested that the Kronecker-Factored Approximate Curvature (\mbox{K-FAC}) method allows for greater scalability to large batch sizes, for non-convex machine learning problems such as neural network optimization, as well as greater robustness to variation in model hyperparameters. Here, we perform a detailed empirical analysis of large batch size training %of these two hypotheses, for both \mbox{K-FAC} and SGD, evaluating performance in terms of both wall-clock time and aggregate computational cost. Our main results are twofold: first, we find that both \mbox{K-FAC} and SGD doesn't have ideal scalability behavior beyond a certain batch size, and that \mbox{K-FAC} does not exhibit improved large-batch scalability behavior, as compared to SGD; and second, we find that \mbox{K-FAC}, in addition to requiring more hyperparameters to tune, suffers from similar hyperparameter sensitivity behavior as does SGD. We discuss extensive results using ResNet and AlexNet on \mbox{CIFAR-10} and SVHN, respectively, as well as more general implications of our findings.
References in corpus (17)
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Don't Decay the Learning Rate, Increase the Batch Size
- On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
- Optimizing Neural Networks with Kronecker-factored Approximate Curvature
- Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
- Measuring the Effects of Data Parallelism on Neural Network Training
- An Empirical Model of Large-Batch Training
- The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning
- Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning
- AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks
- Hessian-based Analysis of Large Batch Training and Robustness to Adversaries
- Sub-Sampled Newton Methods I: Globally Convergent Algorithms
- Sub-Sampled Newton Methods II: Local Convergence Rates
- On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent
- Large batch size training of neural networks with adversarial training and second-order information
- Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networks
- Parameter Re-Initialization through Cyclical Batch Size Schedules