On the Impact of Device and Behavioral Heterogeneity in Federated Learning
arXiv:2102.07500
Abstract
Federated learning (FL) is becoming a popular paradigm for collaborative learning over distributed, private datasets owned by non-trusting entities. FL has seen successful deployment in production environments, and it has been adopted in services such as virtual keyboards, auto-completion, item recommendation, and several IoT applications. However, FL comes with the challenge of performing training over largely heterogeneous datasets, devices, and networks that are out of the control of the centralized FL server. Motivated by this inherent setting, we make a first step towards characterizing the impact of device and behavioral heterogeneity on the trained model. We conduct an extensive empirical study spanning close to 1.5K unique configurations on five popular FL benchmarks. Our analysis shows that these sources of heterogeneity have a major impact on both model performance and fairness, thus sheds light on the importance of considering heterogeneity in FL system design.
References in corpus (18)
- Deep Learning with Differential Privacy
- Federated Learning: Strategies for Improving Communication Efficiency
- Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning
- Federated Learning for Mobile Keyboard Prediction
- Learning Differentially Private Recurrent Language Models
- Revisiting Distributed Synchronous SGD
- Communication-Efficient Federated Deep Learning with Asynchronous Model Update and Temporally Weighted Aggregation
- Horovod: fast and easy distributed deep learning in TensorFlow
- Applied Federated Learning: Improving Google Keyboard Query Suggestions
- Improving Federated Learning Personalization via Model Agnostic Meta Learning
- Revisiting Small Batch Training for Deep Neural Networks
- A generic framework for privacy preserving deep learning
- LEAF: A Benchmark for Federated Settings
- Adaptive Gradient Methods with Dynamic Bound of Learning Rate
- Accelerating Federated Learning over Reliability-Agnostic Clients in Mobile Edge Computing Systems
- AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training
- Characterizing Impacts of Heterogeneity in Federated Learning upon Large-Scale Smartphone Data
- Heterogeneous Data-Aware Federated Learning