EmbRace: Accelerating Sparse Communication for Distributed Training of NLP Neural Networks
arXiv:2110.09132 · doi:10.1145/3545008.3545011
Abstract
Distributed data-parallel training has been widely adopted for deep neural network (DNN) models. Although current deep learning (DL) frameworks scale well for dense models like image classification models, we find that these DL frameworks have relatively low scalability for sparse models like natural language processing (NLP) models that have highly sparse embedding tables. Most existing works overlook the sparsity of model parameters thus suffering from significant but unnecessary communication overhead. In this paper, we propose EmbRace, an efficient communication framework to accelerate communications of distributed training for sparse models. EmbRace introduces Sparsity-aware Hybrid Communication, which integrates AlltoAll and model parallelism into data-parallel training, so as to reduce the communication overhead of highly sparse parameters. To effectively overlap sparse communication with both backward and forward computation, EmbRace further designs a 2D Communication Scheduling approach which optimizes the model computation procedure, relaxes the dependency of embeddings, and schedules the sparse communications of each embedding row with a priority queue. We have implemented a prototype of EmbRace based on PyTorch and Horovod, and conducted comprehensive evaluations with four representative NLP models. Experimental results show that EmbRace achieves up to 2.41X speedup compared to the state-of-the-art distributed training baselines.
References in corpus (5)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- PyTorch Distributed: Experiences on Accelerating Data Parallel Training
- Priority-based Parameter Propagation for Distributed DNN Training
- S2 Reducer: High-Performance Sparse Communication to Accelerate Distributed Deep Learning