Helios: Heterogeneity-Aware Federated Learning with Dynamically Balanced Collaboration
arXiv:1912.01684
Abstract
In this paper, we propose Helios, a heterogeneity-aware FL framework to tackle the straggler issue. Helios identifies individual devices' heterogeneous training capability, and therefore the expected neural network model training volumes regarding the collaborative training pace. For straggling devices, a "soft-training" method is proposed to dynamically compress the original identical training model into the expected volume through a rotating neuron training approach. With extensive algorithm analysis and optimization schemes, the stragglers can be accelerated while retaining the convergence for local training as well as federated collaboration.
arXiv comment6 pages, 7 figures
References in corpus (16)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Federated Learning: Strategies for Improving Communication Efficiency
- Federated Learning with Non-IID Data
- Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning
- Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
- Towards Federated Learning at Scale: System Design
- Communication-Efficient Federated Deep Learning with Asynchronous Model Update and Temporally Weighted Aggregation
- Differentially Private Federated Learning: A Client Level Perspective
- Applied Federated Learning: Improving Google Keyboard Query Suggestions
- Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data
- Asynchronous Federated Optimization
- Expanding the Reach of Federated Learning by Reducing Client Resource Requirements
- Federated Multi-Task Learning
- EdgeSpeechNets: Highly Efficient Deep Neural Networks for Speech Recognition on the Edge
- Distributed Learning of Deep Neural Networks using Independent Subnet Training
- Taming Momentum in a Distributed Asynchronous Environment
Cited by in corpus (5)
- Convergence of Edge Computing and Deep Learning: A Comprehensive Survey
- Adaptive Gradient Sparsification for Efficient Federated Learning: An Online Learning Approach
- Federated Action Recognition on Heterogeneous Embedded Devices
- Critical Learning Periods in Federated Learning
- FLRA: A Reference Architecture for Federated Learning Systems