Communication-Efficient Federated Learning with Compensated Overlap-FedAvg

Abstract

Petabytes of data are generated each day by emerging Internet of Things (IoT), but only few of them can be finally collected and used for Machine Learning (ML) purposes due to the apprehension of data & privacy leakage, which seriously retarding ML's growth. To alleviate this problem, Federated learning is proposed to perform model training by multiple clients' combined data without the dataset sharing within the cluster. Nevertheless, federated learning introduces massive communication overhead as the synchronized data in each epoch is of the same size as the model, and thereby leading to a low communication efficiency. Consequently, variant methods mainly focusing on the communication rounds reduction and data compression are proposed to reduce the communication overhead of federated learning. In this paper, we propose Overlap-FedAvg, a framework that parallels the model training phase with model uploading & downloading phase, so that the latter phase can be totally covered by the former phase. Compared to vanilla FedAvg, Overlap-FedAvg is further developed with a hierarchical computing strategy, a data compensation mechanism and a nesterov accelerated gradients~(NAG) algorithm. Besides, Overlap-FedAvg is orthogonal to many other compression methods so that they can be applied together to maximize the utilization of the cluster. Furthermore, the theoretical analysis is provided to prove the convergence of the proposed Overlap-FedAvg framework. Extensive experiments on both conventional and recurrent tasks with multiple models and datasets also demonstrate that the proposed Overlap-FedAvg framework substantially boosts the federated learning process.

arXiv comment15 pages, 9 figures

References in corpus (18)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Learning with Differential Privacy
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Language Models are Few-Shot Learners
• Federated Learning: Strategies for Improving Communication Efficiency
• Federated Learning with Non-IID Data
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• On the Convergence of FedAvg on Non-IID Data
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• Pointer Sentinel Mixture Models
• Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption
• iDLG: Improved Deep Leakage from Gradients
• Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization
• Asynchronous Stochastic Gradient Descent with Delay Compensation
• Gradient Descent Converges to Minimizers
• Faster On-Device Training Using New Federated Momentum Algorithm

Cited by in corpus (2)

• Federated Dynamic Neural Network for Deep MIMO Detection
• Integrated user scheduling and beam steering in over-the-air federated learning for mobile IoT