ResIST: Layer-Wise Decomposition of ResNets for Distributed Training

Abstract

We propose ResIST, a novel distributed training protocol for Residual Networks (ResNets). ResIST randomly decomposes a global ResNet into several shallow sub-ResNets that are trained independently in a distributed manner for several local iterations, before having their updates synchronized and aggregated into the global model. In the next round, new sub-ResNets are randomly generated and the process repeats until convergence. By construction, per iteration, ResIST communicates only a small portion of network parameters to each machine and never uses the full model during training. Thus, ResIST reduces the per-iteration communication, memory, and time requirements of ResNet training to only a fraction of the requirements of full-model training. In comparison to common protocols, like data-parallel training and data-parallel training with local SGD, ResIST yields a decrease in communication and compute requirements, while being competitive with respect to model performance.

arXiv comment26 pages, 8 figures, pre-print under review

References in corpus (39)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Decoupled Weight Decay Regularization
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Wide Residual Networks
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• Pruning Filters for Efficient ConvNets
• Residual Networks Behave Like Ensembles of Relatively Shallow Networks
• Neural Ordinary Differential Equations
• Horovod: fast and easy distributed deep learning in TensorFlow
• Large Batch Training of Convolutional Networks
• YOLO9000: Better, Faster, Stronger
• Visualizing and Understanding Convolutional Networks
• Do ImageNet Classifiers Generalize to ImageNet?
• Aggregated Residual Transformations for Deep Neural Networks
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• Deep Networks with Stochastic Depth
• Don't Use Large Mini-Batches, Use Local SGD
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication
• Communication Compression for Decentralized Training
• D${}^2$: Decentralized Training over Decentralized Data
• A Unified Theory of Decentralized SGD with Changing Topology and Local Updates
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
• ReZero is All You Need: Fast Convergence at Large Depth
• Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View
• Communication-Efficient Distributed Deep Learning: A Comprehensive Survey
• Parallel SGD: When does averaging help?
• Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform
• High-Accuracy Low-Precision Training
• XPipe: Efficient Pipeline Model Parallelism for Multi-GPU DNN Training
• A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth
• Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs
• Distributed Learning of Deep Neural Networks using Independent Subnet Training
• Layer-Parallel Training of Deep Residual Neural Networks
• Advances in Asynchronous Parallel and Distributed Optimization
• Layered SGD: A Decentralized and Synchronous SGD Algorithm for Scalable Deep Neural Network Training
• LAMP: Large Deep Nets with Automated Model Parallelism for Image Segmentation
• Automatic Model Parallelism for Deep Neural Networks with Compiler and Hardware Support