Pipelined Backpropagation at Scale: Training Large Models without Batches
arXiv:2003.11666
Abstract
New hardware can substantially increase the speed and efficiency of deep neural network training. To guide the development of future hardware architectures, it is pertinent to explore the hardware and machine learning properties of alternative training algorithms. In this work we evaluate the use of small batch, fine-grained Pipelined Backpropagation, an asynchronous pipeline parallel training algorithm that has significant hardware advantages. We introduce two methods, Spike Compensation and Linear Weight Prediction, that effectively mitigate the downsides caused by the asynchronicity of Pipelined Backpropagation and outperform existing techniques in our setting. We show that appropriate normalization and small batch sizes can also aid training. With our methods, fine-grained Pipelined Backpropagation using a batch size of one can match the accuracy of SGD for multiple networks trained on CIFAR-10 and ImageNet. Simple scaling rules allow the use of existing hyperparameters for traditional training without additional tuning.
Proceedings of the 4th MLSys Conference, 2021
References in corpus (23)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- cuDNN: Efficient Primitives for Deep Learning
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- Revisiting Distributed Synchronous SGD
- Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization
- Revisiting Small Batch Training for Deep Neural Networks
- A Survey on Coarse-Grained Reconfigurable Architectures from a Performance Perspective
- PipeDream: Fast and Efficient Pipeline Parallel DNN Training
- Dissecting the Graphcore IPU Architecture via Microbenchmarking
- Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform
- Decoupled Neural Interfaces using Synthetic Gradients
- Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks
- Decoupled Parallel Backpropagation with Convergence Guarantee
- Training Neural Networks Using Features Replay
- Online Normalization for Training Neural Networks
- AMPNet: Asynchronous Model-Parallel Training for Dynamic Neural Networks
- PipeMare: Asynchronous Pipeline Parallel DNN Training
- Taming Momentum in a Distributed Asynchronous Environment
- Fully Decoupled Neural Network Learning Using Delayed Gradients
- At Stability's Edge: How to Adjust Hyperparameters to Preserve Minima Selection in Asynchronous Training of Neural Networks?
- Adaptive Braking for Mitigating Gradient Delay