Stochastic Weight Averaging in Parallel: Large-Batch Training that Generalizes Well
arXiv:2001.02312
Abstract
We propose Stochastic Weight Averaging in Parallel (SWAP), an algorithm to accelerate DNN training. Our algorithm uses large mini-batches to compute an approximate solution quickly and then refines it by averaging the weights of multiple models computed independently and in parallel. The resulting models generalize equally well as those trained with small mini-batches but are produced in a substantially shorter time. We demonstrate the reduction in training time and the good generalization performance of the resulting models on the computer vision datasets CIFAR10, CIFAR100, and ImageNet.
References in corpus (17)
- Improved Regularization of Convolutional Neural Networks with Cutout
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Horovod: fast and easy distributed deep learning in TensorFlow
- Train longer, generalize better: closing the generalization gap in large batch training of neural networks
- Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
- Don't Use Large Mini-Batches, Use Local SGD
- Local SGD Converges Fast and Communicates Little
- A Simple Baseline for Bayesian Uncertainty in Deep Learning
- Measuring the Effects of Data Parallelism on Neural Network Training
- There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average
- An Empirical Model of Large-Batch Training
- Sharp Minima Can Generalize For Deep Nets
- The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning
- AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks
- Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks
- On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent
- SWALP : Stochastic Weight Averaging in Low-Precision Training
Cited by in corpus (5)
- RobustART: Benchmarking Robustness on Architecture Design and Training Techniques
- Consensus Control for Decentralized Deep Learning
- Hierarchical Weight Averaging for Deep Neural Networks
- LocalNewton: Reducing Communication Bottleneck for Distributed Learning
- Loss Landscape Dependent Self-Adjusting Learning Rates in Decentralized Stochastic Gradient Descent