Communication-Efficient Decentralized Learning with Sparsification and Adaptive Peer Selection

Abstract

Distributed learning techniques such as federated learning have enabled multiple workers to train machine learning models together to reduce the overall training time. However, current distributed training algorithms (centralized or decentralized) suffer from the communication bottleneck on multiple low-bandwidth workers (also on the server under the centralized architecture). Although decentralized algorithms generally have lower communication complexity than the centralized counterpart, they still suffer from the communication bottleneck for workers with low network bandwidth. To deal with the communication problem while being able to preserve the convergence performance, we introduce a novel decentralized training algorithm with the following key features: 1) It does not require a parameter server to maintain the model during training, which avoids the communication pressure on any single peer. 2) Each worker only needs to communicate with a single peer at each communication round with a highly compressed model, which can significantly reduce the communication traffic on the worker. We theoretically prove that our sparsification algorithm still preserves convergence properties. 3) Each worker dynamically selects its peer at different communication rounds to better utilize the bandwidth resources. We conduct experiments with convolutional neural networks on 32 workers to verify the effectiveness of our proposed algorithm compared to seven existing methods. Experimental results show that our algorithm significantly reduces the communication traffic and generally select relatively high bandwidth peers.

References in corpus (10)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Federated Learning with Non-IID Data
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Optimal Distributed Online Prediction using Mini-Batches
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• Communication Compression for Decentralized Training
• AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training
• Understanding Top-k Sparsification in Distributed Deep Learning
• A DAG Model of Synchronous Stochastic Gradient Descent in Distributed Deep Learning

Cited by in corpus (5)

• FedML: A Research Library and Benchmark for Federated Machine Learning
• Communication-Efficient Distributed Deep Learning: A Comprehensive Survey
• GFL: A Decentralized Federated Learning Framework Based On Blockchain
• Efficient Ring-topology Decentralized Federated Learning with Deep Generative Models for Industrial Artificial Intelligent
• Communication-efficient Decentralized Machine Learning over Heterogeneous Networks