FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout

Abstract

Federated Learning (FL) has been gaining significant traction across different ML tasks, ranging from vision to keyboard predictions. In large-scale deployments, client heterogeneity is a fact and constitutes a primary problem for fairness, training performance and accuracy. Although significant efforts have been made into tackling statistical data heterogeneity, the diversity in the processing capabilities and network bandwidth of clients, termed as system heterogeneity, has remained largely unexplored. Current solutions either disregard a large portion of available devices or set a uniform limit on the model's capacity, restricted by the least capable participants. In this work, we introduce Ordered Dropout, a mechanism that achieves an ordered, nested representation of knowledge in deep neural networks (DNNs) and enables the extraction of lower footprint submodels without the need of retraining. We further show that for linear maps our Ordered Dropout is equivalent to SVD. We employ this technique, along with a self-distillation methodology, in the realm of FL in a framework called FjORD. FjORD alleviates the problem of client system heterogeneity by tailoring the model width to the client's capabilities. Extensive evaluation on both CNNs and RNNs across diverse modalities shows that FjORD consistently leads to significant performance gains over state-of-the-art baselines, while maintaining its nested structure.

Accepted at the 35th Conference on Neural Information Processing Systems (NeurIPS), 2021

References in corpus (6)

• Distilling the Knowledge in a Neural Network
• Neural Architecture Search with Reinforcement Learning
• FedMD: Heterogenous Federated Learning via Model Distillation
• Expanding the Reach of Federated Learning by Reducing Client Resource Requirements
• Bayesian Nonparametric Federated Learning of Neural Networks
• Federated Learning with Matched Averaging

Cited by in corpus (14)

• A Field Guide to Federated Optimization
• Towards Fairness-Aware Federated Learning
• Topology-aware Federated Learning in Edge Computing: A Comprehensive Survey
• Adaptive Inference through Early-Exit Networks: Design, Challenges and Directions
• Artificial Intelligence of Things: A Survey
• FLrce: Resource-Efficient Federated Learning with Early-Stopping Strategy
• Personalized Federated Learning through Local Memorization
• Efficient and Private Federated Learning with Partially Trainable Networks
• EchoPFL: Asynchronous Personalized Federated Learning on Mobile Devices with On-Demand Staleness Control
• Smart at what cost? Characterising Mobile Deep Neural Networks in the wild
• Energy-Aware Heterogeneous Federated Learning via Approximate DNN Accelerators
• FedHybrid: Breaking the Memory Wall of Federated Learning via Hybrid Tensor Management
• A Two-Timescale Approach for Wireless Federated Learning with Parameter Freezing and Power Control
• Moss: Proxy Model-based Full-Weight Aggregation in Federated Learning with Heterogeneous Models