RingFed: Reducing Communication Costs in Federated Learning on Non-IID Data
arXiv:2107.08873
Abstract
Federated learning is a widely used distributed deep learning framework that protects the privacy of each client by exchanging model parameters rather than raw data. However, federated learning suffers from high communication costs, as a considerable number of model parameters need to be transmitted many times during the training process, making the approach inefficient, especially when the communication network bandwidth is limited. This article proposes RingFed, a novel framework to reduce communication overhead during the training process of federated learning. Rather than transmitting parameters between the center server and each client, as in original federated learning, in the proposed RingFed, the updated parameters are transmitted between each client in turn, and only the final result is transmitted to the central server, thereby reducing the communication overhead substantially. After several local updates, clients first send their parameters to another proximal client, not to the center server directly, to preaggregate. Experiments on two different public datasets show that RingFed has fast convergence, high model accuracy, and low communication cost.
References in corpus (16)
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- Federated Learning: Strategies for Improving Communication Efficiency
- Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
- Optimal Distributed Online Prediction using Mini-Batches
- Federated Optimization in Heterogeneous Networks
- Don't Use Large Mini-Batches, Use Local SGD
- Local SGD Converges Fast and Communicates Little
- Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
- Expanding the Reach of Federated Learning by Reducing Client Resource Requirements
- Adaptive Federated Optimization
- Federated Multi-Task Learning
- First Analysis of Local GD on Heterogeneous Data
- A Communication Efficient Collaborative Learning Framework for Distributed Features
- Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGD
- Gradient Descent with Compressed Iterates