Expanding the Reach of Federated Learning by Reducing Client Resource Requirements

Abstract

Communication on heterogeneous edge networks is a fundamental bottleneck in Federated Learning (FL), restricting both model capacity and user participation. To address this issue, we introduce two novel strategies to reduce communication costs: (1) the use of lossy compression on the global model sent server-to-client; and (2) Federated Dropout, which allows users to efficiently train locally on smaller subsets of the global model and also provides a reduction in both client-to-server communication and local computation. We empirically show that these strategies, combined with existing compression approaches for client-to-server communication, collectively provide up to a $14\times$ reduction in server-to-client communication, a $1.7\times$ reduction in local computation, and a $28\times$ reduction in upload communication, all without degrading the quality of the final model. We thus comprehensively reduce FL's impact on client device resources, allowing higher capacity models to be trained, and a more diverse set of users to be reached.

References in corpus (4)

• Distilling the Knowledge in a Neural Network
• Federated Optimization: Distributed Machine Learning for On-Device Intelligence
• EMNIST: an extension of MNIST to handwritten letters
• AIDE: Fast and Communication Efficient Distributed Optimization

Cited by in corpus (31)

• Towards Federated Learning at Scale: System Design
• Active Federated Learning
• Robust Federated Learning: The Case of Affine Distribution Shifts
• Throughput-Optimal Topology Design for Cross-Silo Federated Learning
• Privacy for Free: Communication-Efficient Learning with Differential Privacy Using Sketches
• Federated User Representation Learning
• Faster On-Device Training Using New Federated Momentum Algorithm
• Fairness and Accuracy in Federated Learning
• Secure Federated Submodel Learning
• FedJAX: Federated learning simulation with JAX
• Real-time Federated Evolutionary Neural Architecture Search
• FedCon: A Contrastive Framework for Federated Semi-Supervised Learning
• Efficient and Private Federated Learning with Partially Trainable Networks
• From Federated Learning to Federated Neural Architecture Search: A Survey
• Adaptive Federated Dropout: Improving Communication Efficiency and Generalization for Federated Learning
• RingFed: Reducing Communication Costs in Federated Learning on Non-IID Data
• Federated Mixture of Experts
• Federated Learning on Non-IID Data: A Survey
• Distributed Fixed Point Methods with Compressed Iterates
• FLFE: A Communication-Efficient and Privacy-Preserving Federated Feature Engineering Framework
• AdaptCL: Efficient Collaborative Learning with Dynamic and Adaptive Pruning
• FedPrune: Towards Inclusive Federated Learning
• Critical Learning Periods in Federated Learning
• Communication Optimization in Large Scale Federated Learning using Autoencoder Compressed Weight Updates
• TiFL: A Tier-based Federated Learning System
• ADDS: Adaptive Differentiable Sampling for Robust Multi-Party Learning
• Design and Analysis of Uplink and Downlink Communications for Federated Learning
• Budgeted Online Selection of Candidate IoT Clients to Participate in Federated Learning
• Wireless Distributed Edge Learning: How Many Edge Devices Do We Need?
• CatFedAvg: Optimising Communication-efficiency and Classification Accuracy in Federated Learning
• Distantly Supervised Relation Extraction in Federated Settings