Parallel Training of Deep Networks with Local Updates

Abstract

Deep learning models trained on large data sets have been widely successful in both vision and language domains. As state-of-the-art deep learning architectures have continued to grow in parameter count so have the compute budgets and times required to train them, increasing the need for compute-efficient methods that parallelize training. Two common approaches to parallelize the training of deep networks have been data and model parallelism. While useful, data and model parallelism suffer from diminishing returns in terms of compute efficiency for large batch sizes. In this paper, we investigate how to continue scaling compute efficiently beyond the point of diminishing returns for large batches through local parallelism, a framework which parallelizes training of individual layers in deep networks by replacing global backpropagation with truncated layer-wise backpropagation. Local parallelism enables fully asynchronous layer-wise parallelism with a low memory footprint, and requires little communication overhead compared with model parallelism. We show results in both vision and language domains across a diverse set of architectures, and find that local parallelism is particularly effective in the high-compute regime.

arXiv commentFirst two authors - Michael Laskin and Luke Metz - contributed equally. Order was determined by a coin flip

References in corpus (30)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Distilling the Knowledge in a Neural Network
• The NumPy array: a structure for efficient numerical computation
• A Simple Framework for Contrastive Learning of Visual Representations
• Language Models are Few-Shot Learners
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Scaling Laws for Neural Language Models
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Revisiting Distributed Synchronous SGD
• Horovod: fast and easy distributed deep learning in TensorFlow
• Revisiting Small Batch Training for Deep Neural Networks
• Gradient Estimation Using Stochastic Computation Graphs
• Unsupervised Learning by Competing Hidden Units
• Measuring the Effects of Data Parallelism on Neural Network Training
• Distributed Deep Learning Using Synchronous Stochastic Gradient Descent
• An Empirical Model of Large-Batch Training
• PipeDream: Fast and Efficient Pipeline Parallel DNN Training
• Meta-Learning Update Rules for Unsupervised Representation Learning
• Dissecting the Graphcore IPU Architecture via Microbenchmarking
• Putting An End to End-to-End: Gradient-Isolated Learning of Representations
• Decoupled Parallel Backpropagation with Convergence Guarantee
• Training Neural Networks Using Features Replay
• Understanding and correcting pathologies in the training of learned optimizers
• PipeMare: Asynchronous Pipeline Parallel DNN Training
• LoCo: Local Contrastive Representation Learning
• Using a thousand optimization tasks to learn hyperparameter search strategies
• Local Unsupervised Learning for Image Analysis
• Bio-Inspired Hashing for Unsupervised Similarity Search