Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networks

Abstract

Large-scale distributed training of deep neural networks suffer from the generalization gap caused by the increase in the effective mini-batch size. Previous approaches try to solve this problem by varying the learning rate and batch size over epochs and layers, or some ad hoc modification of the batch normalization. We propose an alternative approach using a second-order optimization method that shows similar generalization capability to first-order methods, but converges faster and can handle larger mini-batches. To test our method on a benchmark where highly optimized first-order methods are available as references, we train ResNet-50 on ImageNet. We converged to 75% Top-1 validation accuracy in 35 epochs for mini-batch sizes under 16,384, and achieved 75% even with a mini-batch size of 131,072, which took only 978 iterations.

arXiv comment10 pages, 7 figures. Accepted at CVPR 2019, Long Beach, CA

References in corpus (15)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Don't Decay the Learning Rate, Increase the Batch Size
• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes
• Don't Use Large Mini-Batches, Use Local SGD
• L2 Regularization versus Batch and Weight Normalization
• Measuring the Effects of Data Parallelism on Neural Network Training
• AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks
• Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash
• ChainerMN: Scalable Distributed Deep Learning Framework
• Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach
• Practical Gauss-Newton Optimisation for Deep Learning

Cited by in corpus (9)

• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
• Optimization for deep learning: theory and algorithms
• Augment your batch: better training with larger batches
• Large batch size training of neural networks with adversarial training and second-order information
• Orchestrating the Development Lifecycle of Machine Learning-Based IoT Applications: A Taxonomy and Survey
• Tractable structured natural gradient descent using local parameterizations
• Fast Approximation of the Gauss-Newton Hessian Matrix for the Multilayer Perceptron
• Inefficiency of K-FAC for Large Batch Size Training
• Concurrent Adversarial Learning for Large-Batch Training