FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout

Abstract

Federated Learning (FL) has been gaining significant traction across different ML tasks, ranging from vision to keyboard predictions. In large-scale deployments, client heterogeneity is a fact and constitutes a primary problem for fairness, training performance and accuracy. Although significant efforts have been made into tackling statistical data heterogeneity, the diversity in the processing capabilities and network bandwidth of clients, termed as system heterogeneity, has remained largely unexplored. Current solutions either disregard a large portion of available devices or set a uniform limit on the model's capacity, restricted by the least capable participants. In this work, we introduce Ordered Dropout, a mechanism that achieves an ordered, nested representation of knowledge in deep neural networks (DNNs) and enables the extraction of lower footprint submodels without the need of retraining. We further show that for linear maps our Ordered Dropout is equivalent to SVD. We employ this technique, along with a self-distillation methodology, in the realm of FL in a framework called FjORD. FjORD alleviates the problem of client system heterogeneity by tailoring the model width to the client's capabilities. Extensive evaluation on both CNNs and RNNs across diverse modalities shows that FjORD consistently leads to significant performance gains over state-of-the-art baselines, while maintaining its nested structure.

arXiv commentAccepted at the 35th Conference on Neural Information Processing Systems (NeurIPS), 2021

References in corpus (21)

• Distilling the Knowledge in a Neural Network
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Neural Architecture Search with Reinforcement Learning
• Federated Learning: Strategies for Improving Communication Efficiency
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• How To Backdoor Federated Learning
• Learning Differentially Private Recurrent Language Models
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Differentially Private Federated Learning: A Client Level Perspective
• FedMD: Heterogenous Federated Learning via Model Distillation
• LEAF: A Benchmark for Federated Settings
• ATOMO: Communication-efficient Learning via Atomic Sparsification
• Expanding the Reach of Federated Learning by Reducing Client Resource Requirements
• Bayesian Nonparametric Federated Learning of Neural Networks
• Think Locally, Act Globally: Federated Learning with Local and Global Representations
• Flower: A Friendly Federated Learning Research Framework
• Federated Learning with Matched Averaging
• Balanced Sparsity for Efficient DNN Inference on GPU
• Natural Compression for Distributed Deep Learning
• Learning Ordered Representations with Nested Dropout
• Federated Learning With Quantized Global Model Updates

Cited by in corpus (14)

• A Field Guide to Federated Optimization
• Towards Fairness-Aware Federated Learning
• Topology-aware Federated Learning in Edge Computing: A Comprehensive Survey
• Adaptive Inference through Early-Exit Networks: Design, Challenges and Directions
• Artificial Intelligence of Things: A Survey
• FLrce: Resource-Efficient Federated Learning with Early-Stopping Strategy
• Personalized Federated Learning through Local Memorization
• Efficient and Private Federated Learning with Partially Trainable Networks
• EchoPFL: Asynchronous Personalized Federated Learning on Mobile Devices with On-Demand Staleness Control
• Smart at what cost? Characterising Mobile Deep Neural Networks in the wild
• Energy-Aware Heterogeneous Federated Learning via Approximate DNN Accelerators
• FedHybrid: Breaking the Memory Wall of Federated Learning via Hybrid Tensor Management
• A Two-Timescale Approach for Wireless Federated Learning with Parameter Freezing and Power Control
• Moss: Proxy Model-based Full-Weight Aggregation in Federated Learning with Heterogeneous Models