Distributed Learning of Deep Neural Networks using Independent Subnet Training

Abstract

Distributed machine learning (ML) can bring more computational resources to bear than single-machine learning, thus enabling reductions in training time. Distributed learning partitions models and data over many machines, allowing model and dataset sizes beyond the available compute power and memory of a single machine. In practice though, distributed ML is challenging when distribution is mandatory, rather than chosen by the practitioner. In such scenarios, data could unavoidably be separated among workers due to limited memory capacity per worker or even because of data privacy issues. There, existing distributed methods will utterly fail due to dominant transfer costs across workers, or do not even apply. We propose a new approach to distributed fully connected neural network learning, called independent subnet training (IST), to handle these cases. In IST, the original network is decomposed into a set of narrow subnetworks with the same depth. These subnetworks are then trained locally before parameters are exchanged to produce new subnets and the training cycle repeats. Such a naturally "model parallel" approach limits memory usage by storing only a portion of network parameters on each device. Additionally, no requirements exist for sharing data between workers (i.e., subnet training is local and independent) and communication volume and frequency are reduced by decomposing the original network into independent subnets. These properties of IST can cope with issues due to distributed data, slow interconnects, or limited device memory, making IST a suitable approach for cases of mandatory distribution. We show experimentally that IST results in training times that are much lower than common distributed learning approaches.

References in corpus (23)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• ADADELTA: An Adaptive Learning Rate Method
• Federated Optimization: Distributed Machine Learning for On-Device Intelligence
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
• One weird trick for parallelizing convolutional neural networks
• Revisiting Distributed Synchronous SGD
• Don't Decay the Learning Rate, Increase the Batch Size
• Horovod: fast and easy distributed deep learning in TensorFlow
• Large Batch Training of Convolutional Networks
• Don't Use Large Mini-Batches, Use Local SGD
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
• PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization
• Parallel SGD: When does averaging help?
• On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent
• Scale out for large minibatch SGD: Residual network training on ImageNet-1K with improved accuracy and reduced time to train
• MLSys: The New Frontier of Machine Learning Systems
• On the Ineffectiveness of Variance Reduced Optimization for Deep Learning
• Large-Batch Training for LSTM and Beyond
• Gradient Descent with Compressed Iterates
• GPU Accelerated Sub-Sampled Newton's Method
• Inefficiency of K-FAC for Large Batch Size Training

Cited by in corpus (5)

• Helios: Heterogeneity-Aware Federated Learning with Dynamically Balanced Collaboration
• Collate: Collaborative Neural Network Learning for Latency-Critical Edge Systems
• ResIST: Layer-Wise Decomposition of ResNets for Distributed Training
• Machine Learning Systems for Intelligent Services in the IoT: A Survey
• Masked Training of Neural Networks with Partial Gradients