Large batch size training of neural networks with adversarial training and second-order information

Abstract

The most straightforward method to accelerate Stochastic Gradient Descent (SGD) computation is to distribute the randomly selected batch of inputs over multiple processors. To keep the distributed processors fully utilized requires commensurately growing the batch size. However, large batch training often leads to poorer generalization. A recently proposed solution for this problem is to use adaptive batch sizes in SGD. In this case, one starts with a small number of processes and scales the processes as training progresses. Two major challenges with this approach are (i) that dynamically resizing the cluster can add non-trivial overhead, in part since it is currently not supported, and (ii) that the overall speed up is limited by the initial phase with smaller batches. In this work, we address both challenges by developing a new adaptive batch size framework, with autoscaling based on the Ray framework. This allows very efficient elastic scaling with negligible resizing overhead (0.32\% of time for ResNet18 ImageNet training). Furthermore, we propose a new adaptive batch size training scheme using second order methods and adversarial training. These enable increasing batch sizes earlier during training, which leads to better training time. We extensively evaluate our method on Cifar-10/100, SVHN, TinyImageNet, and ImageNet datasets, using multiple neural networks, including ResNets and smaller networks such as SqueezeNext. Our method exceeds the performance of existing solutions in terms of both accuracy and the number of SGD iterations (up to 1\% and $5\times$, respectively). Importantly, this is achieved without any additional hyper-parameter tuning to tailor our method in any of these experiments.

References in corpus (23)

• Explaining and Harnessing Adversarial Examples
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Wide Residual Networks
• Don't Decay the Learning Rate, Increase the Batch Size
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Large Batch Training of Convolutional Networks
• Adversarial Examples Are Not Bugs, They Are Features
• Understanding Adversarial Training: Increasing Local Stability of Neural Nets through Robust Optimization
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• No More Pesky Learning Rates
• Measuring the Effects of Data Parallelism on Neural Network Training
• An Empirical Model of Large-Batch Training
• Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning
• AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks
• Image Classification at Supercomputer Scale
• Hessian-based Analysis of Large Batch Training and Robustness to Adversaries
• Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash
• A Progressive Batching L-BFGS Method for Machine Learning
• On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent
• PyHessian: Neural Networks Through the Lens of the Hessian
• Coupling Adaptive Batch Sizes with Learning Rates
• Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networks
• Parallel Stochastic Gradient Descent with Sound Combiners

Cited by in corpus (18)

• Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash
• HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks
• HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision
• PyHessian: Neural Networks Through the Lens of the Hessian
• Deep Learning for Surface Wave Identification in Distributed Acoustic Sensing Data
• On the Utility of Gradient Compression in Distributed Training Systems
• Accordion: Adaptive Gradient Communication via Critical Learning Regime Identification
• Taxonomizing local versus global structure in neural network loss landscapes
• Study on the Large Batch Size Training of Neural Networks Based on the Second Order Gradient
• Trust Region Based Adversarial Attack on Neural Networks
• Scavenger: A Cloud Service for Optimizing Cost and Performance of ML Training
• Topology Aware Deep Learning for Wireless Network Optimization
• Accelerating Distributed ML Training via Selective Synchronization
• Parameter Re-Initialization through Cyclical Batch Size Schedules
• Inefficiency of K-FAC for Large Batch Size Training
• Concurrent Adversarial Learning for Large-Batch Training
• Multi-Agent Semi-Siamese Training for Long-tail and Shallow Face Learning
• HyperSched: Dynamic Resource Reallocation for Model Development on a Deadline