Concurrent Adversarial Learning for Large-Batch Training

Abstract

Large-batch training has become a commonly used technique when training neural networks with a large number of GPU/TPU processors. As batch size increases, stochastic optimizers tend to converge to sharp local minima, leading to degraded test performance. Current methods usually use extensive data augmentation to increase the batch size, but we found the performance gain with data augmentation decreases as batch size increases, and data augmentation will become insufficient after certain point. In this paper, we propose to use adversarial learning to increase the batch size in large-batch training. Despite being a natural choice for smoothing the decision surface and biasing towards a flat region, adversarial learning has not been successfully applied in large-batch training since it requires at least two sequential gradient computations at each step, which will at least double the running time compared with vanilla training even with a large number of processors. To overcome this issue, we propose a novel Concurrent Adversarial Learning (ConAdv) method that decouple the sequential gradient computations in adversarial learning by utilizing staled parameters. Experimental results demonstrate that ConAdv can successfully increase the batch size on ResNet-50 training on ImageNet while maintaining high accuracy. In particular, we show ConAdv along can achieve 75.3\% top-1 accuracy on ImageNet ResNet-50 training with 96K batch size, and the accuracy can be further improved to 76.2\% when combining ConAdv with data augmentation. This is the first work successfully scales ResNet-50 training batch size to 96K.

arXiv commentAccepted to ICLR 2022

References in corpus (21)

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples
• Don't Decay the Learning Rate, Increase the Batch Size
• Large Batch Training of Convolutional Networks
• Fast is better than free: Revisiting adversarial training
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• Understanding Adversarial Training: Increasing Local Stability of Neural Nets through Robust Optimization
• Certifying Some Distributional Robustness with Principled Adversarial Training
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes
• MLPerf Training Benchmark
• Measuring the Effects of Data Parallelism on Neural Network Training
• AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks
• Image Classification at Supercomputer Scale
• You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle
• Yet Another Accelerated SGD: ResNet-50 Training on ImageNet in 74.7 seconds
• Characterizing the Decision Boundary of Deep Neural Networks
• Scale out for large minibatch SGD: Residual network training on ImageNet-1K with improved accuracy and reduced time to train
• Large batch size training of neural networks with adversarial training and second-order information
• Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networks