Characterizing Impacts of Heterogeneity in Federated Learning upon Large-Scale Smartphone Data

Abstract

Federated learning (FL) is an emerging, privacy-preserving machine learning paradigm, drawing tremendous attention in both academia and industry. A unique characteristic of FL is heterogeneity, which resides in the various hardware specifications and dynamic states across the participating devices. Theoretically, heterogeneity can exert a huge influence on the FL training process, e.g., causing a device unavailable for training or unable to upload its model updates. Unfortunately, these impacts have never been systematically studied and quantified in existing FL literature. In this paper, we carry out the first empirical study to characterize the impacts of heterogeneity in FL. We collect large-scale data from 136k smartphones that can faithfully reflect heterogeneity in real-world settings. We also build a heterogeneity-aware FL platform that complies with the standard FL protocol but with heterogeneity in consideration. Based on the data and the platform, we conduct extensive experiments to compare the performance of state-of-the-art FL algorithms under heterogeneity-aware and heterogeneity-unaware settings. Results show that heterogeneity causes non-trivial performance degradation in FL, including up to 9.2% accuracy drop, 2.32x lengthened training time, and undermined fairness. Furthermore, we analyze potential impact factors and find that device failure and participant bias are two potential factors for performance degradation. Our study provides insightful implications for FL practitioners. On the one hand, our findings suggest that FL algorithm designers consider necessary heterogeneity during the evaluation. On the other hand, our findings urge system providers to design specific mechanisms to mitigate the impacts of heterogeneity.

References in corpus (26)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• Neural Architecture Search: A Survey
• Federated Optimization: Distributed Machine Learning for On-Device Intelligence
• Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Federated Learning for Mobile Keyboard Prediction
• How To Backdoor Federated Learning
• Learning Differentially Private Recurrent Language Models
• Revisiting Distributed Synchronous SGD
• Communication-Efficient Federated Deep Learning with Asynchronous Model Update and Temporally Weighted Aggregation
• Applied Federated Learning: Improving Google Keyboard Query Suggestions
• Improving Federated Learning Personalization via Model Agnostic Meta Learning
• FedML: A Research Library and Benchmark for Federated Machine Learning
• FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization
• A generic framework for privacy preserving deep learning
• LEAF: A Benchmark for Federated Settings
• Agnostic Federated Learning
• A Survey on Neural Architecture Search
• Distributed Mean Estimation with Limited Communication
• signSGD with Majority Vote is Communication Efficient And Fault Tolerant
• AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training
• Mining Behavioral Patterns from Millions of Android Users
• Hierarchical Federated Learning through LAN-WAN Orchestration
• Secure Federated Submodel Learning

Cited by in corpus (5)

• Towards Flexible Device Participation in Federated Learning
• Hierarchical Federated Learning through LAN-WAN Orchestration
• On the Impact of Device and Behavioral Heterogeneity in Federated Learning
• More Industry-friendly: Federated Learning with High Efficient Design
• EasyFL: A Low-code Federated Learning Platform For Dummies