Federated Learning for Computationally-Constrained Heterogeneous Devices: A Survey

Abstract

With an increasing number of smart devices like internet of things (IoT) devices deployed in the field, offloadingtraining of neural networks (NNs) to a central server becomes more and more infeasible. Recent efforts toimprove users' privacy have led to on-device learning emerging as an alternative. However, a model trainedonly on a single device, using only local data, is unlikely to reach a high accuracy. Federated learning (FL)has been introduced as a solution, offering a privacy-preserving trade-off between communication overheadand model accuracy by sharing knowledge between devices but disclosing the devices' private data. Theapplicability and the benefit of applying baseline FL are, however, limited in many relevant use cases dueto the heterogeneity present in such environments. In this survey, we outline the heterogeneity challengesFL has to overcome to be widely applicable in real-world applications. We especially focus on the aspect ofcomputation heterogeneity among the participating devices and provide a comprehensive overview of recentworks on heterogeneity-aware FL. We discuss two groups: works that adapt the NN architecture and worksthat approach heterogeneity on a system level, covering Federated Averaging (FedAvg), distillation, and splitlearning-based approaches, as well as synchronous and asynchronous aggregation schemes.

References in corpus (24)

• Distilling the Knowledge in a Neural Network
• Federated Learning: Challenges, Methods, and Future Directions
• Revisiting Distributed Synchronous SGD
• EMNIST: an extension of MNIST to handwritten letters
• Applied Federated Learning: Improving Google Keyboard Query Suggestions
• Asynchronous Federated Optimization
• Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systems
• Hardware Approximate Techniques for Deep Neural Network Accelerators: A Survey
• Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer
• Federated Learning with Differential Privacy: Algorithms and Performance Analysis
• Federated Mutual Learning
• HHHFL: Hierarchical Heterogeneous Horizontal Federated Learning for Electroencephalography
• Model Pruning Enables Efficient Federated Learning on Edge Devices
• A Federated Learning Framework for Healthcare IoT devices
• Asynchronous Federated Learning on Heterogeneous Devices: A Survey
• FedHealth: A Federated Transfer Learning Framework for Wearable Healthcare
• FedHe: Heterogeneous Models and Communication-Efficient Federated Learning
• Papaya: Practical, Private, and Scalable Federated Learning
• Federated Learning for Resource-Constrained IoT Devices: Panoramas and State-of-the-art
• Straggler-Resilient Federated Learning: Leveraging the Interplay Between Statistical Accuracy and System Heterogeneity
• AdaSplit: Adaptive Trade-offs for Resource-constrained Distributed Deep Learning
• FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorization
• CoCoFL: Communication- and Computation-Aware Federated Learning via Partial NN Freezing and Quantization
• Towards Efficient Scheduling of Federated Mobile Devices under Computational and Statistical Heterogeneity

Cited by in corpus (5)

• Advances in Robust Federated Learning: A Survey with Heterogeneity Considerations
• EchoPFL: Asynchronous Personalized Federated Learning on Mobile Devices with On-Demand Staleness Control
• Flotilla: A scalable, modular and resilient federated learning framework for heterogeneous resources
• Energy-Aware Heterogeneous Federated Learning via Approximate DNN Accelerators
• FedDCT: A Dynamic Cross-Tier Federated Learning Framework in Wireless Networks