Expanding the Reach of Federated Learning by Reducing Client Resource Requirements

Abstract

Communication on heterogeneous edge networks is a fundamental bottleneck in Federated Learning (FL), restricting both model capacity and user participation. To address this issue, we introduce two novel strategies to reduce communication costs: (1) the use of lossy compression on the global model sent server-to-client; and (2) Federated Dropout, which allows users to efficiently train locally on smaller subsets of the global model and also provides a reduction in both client-to-server communication and local computation. We empirically show that these strategies, combined with existing compression approaches for client-to-server communication, collectively provide up to a $14\times$ reduction in server-to-client communication, a $1.7\times$ reduction in local computation, and a $28\times$ reduction in upload communication, all without degrading the quality of the final model. We thus comprehensively reduce FL's impact on client device resources, allowing higher capacity models to be trained, and a more diverse set of users to be reached.

References in corpus (11)

• Distilling the Knowledge in a Neural Network
• Federated Learning: Strategies for Improving Communication Efficiency
• Federated Optimization: Distributed Machine Learning for On-Device Intelligence
• Binarized Neural Networks
• EMNIST: an extension of MNIST to handwritten letters
• ATOMO: Communication-efficient Learning via Atomic Sparsification
• Distributed Mean Estimation with Limited Communication
• AIDE: Fast and Communication Efficient Distributed Optimization
• Uncertainty Principles and Vector Quantization
• High-Accuracy Low-Precision Training
• Randomized Distributed Mean Estimation: Accuracy vs Communication

Cited by in corpus (65)

• Convergence of Edge Computing and Deep Learning: A Comprehensive Survey
• Towards Federated Learning at Scale: System Design
• Improving Federated Learning Personalization via Model Agnostic Meta Learning
• From Distributed Machine Learning to Federated Learning: A Survey
• Federated Learning in Mobile Edge Networks: A Comprehensive Survey
• Think Locally, Act Globally: Federated Learning with Local and Global Representations
• Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications
• Edge Intelligence: Architectures, Challenges, and Applications
• Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
• Active Federated Learning
• Artificial Intelligence of Things: A Survey
• Federated Learning for Healthcare Informatics
• FedSAE: A Novel Self-Adaptive Federated Learning Framework in Heterogeneous Systems
• Robust Federated Learning: The Case of Affine Distribution Shifts
• FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout
• Federated Learning with Unbiased Gradient Aggregation and Controllable Meta Updating
• Throughput-Optimal Topology Design for Cross-Silo Federated Learning
• Privacy for Free: Communication-Efficient Learning with Differential Privacy Using Sketches
• Tighter Theory for Local SGD on Identical and Heterogeneous Data
• Federated User Representation Learning
• Fairness and Accuracy in Federated Learning
• Faster On-Device Training Using New Federated Momentum Algorithm
• Towards Flexible Device Participation in Federated Learning
• Breaking the Communication-Privacy-Accuracy Trilemma
• Federated Learning With Quantized Global Model Updates
• Hierarchical Federated Learning through LAN-WAN Orchestration
• Federated Learning with Additional Mechanisms on Clients to Reduce Communication Costs
• Secure Federated Submodel Learning
• TornadoAggregate: Accurate and Scalable Federated Learning via the Ring-Based Architecture
• FedJAX: Federated learning simulation with JAX
• Helios: Heterogeneity-Aware Federated Learning with Dynamically Balanced Collaboration
• Real-time Federated Evolutionary Neural Architecture Search
• Improving Semi-supervised Federated Learning by Reducing the Gradient Diversity of Models
• HFEL: Joint Edge Association and Resource Allocation for Cost-Efficient Hierarchical Federated Edge Learning
• FedCon: A Contrastive Framework for Federated Semi-Supervised Learning
• Gradient Descent with Compressed Iterates
• Efficient and Private Federated Learning with Partially Trainable Networks
• From Federated Learning to Federated Neural Architecture Search: A Survey
• DRIVE: One-bit Distributed Mean Estimation
• FedSiam: Towards Adaptive Federated Semi-Supervised Learning
• Federated Mixture of Experts
• RingFed: Reducing Communication Costs in Federated Learning on Non-IID Data
• Adaptive Gradient Sparsification for Efficient Federated Learning: An Online Learning Approach
• Adaptive Federated Dropout: Improving Communication Efficiency and Generalization for Federated Learning
• Loss Tolerant Federated Learning
• Federated Learning on Non-IID Data: A Survey
• Distributed Fixed Point Methods with Compressed Iterates
• Communication Efficient Federated Learning with Energy Awareness over Wireless Networks
• Trends and Advancements in Deep Neural Network Communication
• Communication Optimization in Large Scale Federated Learning using Autoencoder Compressed Weight Updates
• FLFE: A Communication-Efficient and Privacy-Preserving Federated Feature Engineering Framework
• AdaptCL: Efficient Collaborative Learning with Dynamic and Adaptive Pruning
• Critical Learning Periods in Federated Learning
• FedPrune: Towards Inclusive Federated Learning
• TiFL: A Tier-based Federated Learning System
• FedDropoutAvg: Generalizable federated learning for histopathology image classification
• An Expectation-Maximization Perspective on Federated Learning
• ADDS: Adaptive Differentiable Sampling for Robust Multi-Party Learning
• Communication-Censored Distributed Stochastic Gradient Descent
• Distantly Supervised Relation Extraction in Federated Settings
• CatFedAvg: Optimising Communication-efficiency and Classification Accuracy in Federated Learning
• Budgeted Online Selection of Candidate IoT Clients to Participate in Federated Learning
• Wireless Distributed Edge Learning: How Many Edge Devices Do We Need?
• MyMigrationBot: A Cloud-based Facebook Social Chatbot for Migrant Populations
• Design and Analysis of Uplink and Downlink Communications for Federated Learning