Federated Learning on Non-IID Data Silos: An Experimental Study

Abstract

Due to the increasing privacy concerns and data regulations, training data have been increasingly fragmented, forming distributed databases of multiple "data silos" (e.g., within different organizations and countries). To develop effective machine learning services, there is a must to exploit data from such distributed databases without exchanging the raw data. Recently, federated learning (FL) has been a solution with growing interests, which enables multiple parties to collaboratively train a machine learning model without exchanging their local data. A key and common challenge on distributed databases is the heterogeneity of the data distribution among the parties. The data of different parties are usually non-independently and identically distributed (i.e., non-IID). There have been many FL algorithms to address the learning effectiveness under non-IID data settings. However, there lacks an experimental study on systematically understanding their advantages and disadvantages, as previous studies have very rigid data partitioning strategies among parties, which are hardly representative and thorough. In this paper, to help researchers better understand and study the non-IID data setting in federated learning, we propose comprehensive data partitioning strategies to cover the typical non-IID data cases. Moreover, we conduct extensive experiments to evaluate state-of-the-art FL algorithms. We find that non-IID does bring significant challenges in learning accuracy of FL algorithms, and none of the existing state-of-the-art FL algorithms outperforms others in all cases. Our experiments provide insights for future studies of addressing the challenges in "data silos".

References in corpus (28)

• Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection
• Federated Learning for Mobile Keyboard Prediction
• Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
• Secure Federated Transfer Learning
• Ensemble Distillation for Robust Model Fusion in Federated Learning
• Differentially Private Federated Learning: A Client Level Perspective
• Neo: A Learned Query Optimizer
• FedML: A Research Library and Benchmark for Federated Machine Learning
• FedBN: Federated Learning on Non-IID Features via Local Batch Normalization
• LEAF: A Benchmark for Federated Settings
• Agnostic Federated Learning
• Personalized Federated Learning with Moreau Envelopes
• Benchmarking Learned Indexes
• Federated Adversarial Domain Adaptation
• Bayesian Nonparametric Federated Learning of Neural Networks
• One-Shot Federated Learning
• Adaptive Federated Optimization
• Federated Learning Based on Dynamic Regularization
• Federated Learning with Matched Averaging
• Lower Bounds and Optimal Algorithms for Personalized Federated Learning
• The OARF Benchmark Suite: Characterization and Implications for Federated Learning Systems
• Federated Learning with Only Positive Labels
• Attribution Modeling Increases Efficiency of Bidding in Display Advertising
• Distributionally Robust Federated Averaging
• Evaluation Framework For Large-scale Federated Learning

Cited by in corpus (13)

• FedLab: A Flexible Federated Learning Framework
• The OARF Benchmark Suite: Characterization and Implications for Federated Learning Systems
• No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data
• Model-Contrastive Federated Learning
• Federated Learning for Smart Healthcare: A Survey
• Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning
• Federated Learning on Non-IID Data: A Survey
• Towards Practical Few-shot Federated NLP
• SPATL: Salient Parameter Aggregation and Transfer Learning for Heterogeneous Clients in Federated Learning
• Is Shapley Value fair? Improving Client Selection for Mavericks in Federated Learning
• FedSC: Federated Learning with Semantic-Aware Collaboration
• Personalized Federated Learning over non-IID Data for Indoor Localization
• AutoFL: Enabling Heterogeneity-Aware Energy Efficient Federated Learning