Demystifying the Effects of Non-Independence in Federated Learning

Abstract

Federated Learning (FL) enables statistical models to be built on user-generated data without compromising data security and user privacy. For this reason, FL is well suited for on-device learning from mobile devices where data is abundant and highly privatized. Constrained by the temporal availability of mobile devices, only a subset of devices is accessible to participate in the iterative protocol consisting of training and aggregation. In this study, we take a step toward better understanding the effect of non-independent data distributions arising from block-cyclic sampling. By conducting extensive experiments on visual classification, we measure the effects of block-cyclic sampling (both standalone and in combination with non-balanced block distributions). Specifically, we measure the alterations induced by block-cyclic sampling from the perspective of accuracy, fairness, and convergence rate. Experimental results indicate robustness to cycling over a two-block structure, e.g., due to time zones. In contrast, drawing data samples dependently from a multi-block structure significantly degrades the performance and rate of convergence by up to 26%. Moreover, we find that this performance degeneration is further aggravated by unbalanced block distributions to a point that can no longer be adequately compensated by higher communication and more frequent synchronization.

arXiv comment8 pages, 7 figures

References in corpus (24)

• Overcoming catastrophic forgetting in neural networks
• Federated Learning with Non-IID Data
• Federated Learning for Mobile Keyboard Prediction
• On the Convergence of FedAvg on Non-IID Data
• Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
• Don't Decay the Learning Rate, Increase the Batch Size
• Federated Learning with Personalization Layers
• Applied Federated Learning: Improving Google Keyboard Query Suggestions
• Federated Optimization in Heterogeneous Networks
• Personalized Federated Learning: A Meta-Learning Approach
• Fair Resource Allocation in Federated Learning
• Federated Meta-Learning with Fast Convergence and Efficient Communication
• Federated Learning of a Mixture of Global and Local Models
• Overcoming Forgetting in Federated Learning on Non-IID Data
• Variational Federated Multi-Task Learning
• Federated Multi-Task Learning
• Fairness and Accuracy in Federated Learning
• Fed-Focal Loss for imbalanced data classification in Federated Learning
• Semi-Cyclic Stochastic Gradient Descent
• Specialized federated learning using a mixture of experts
• FedMAX: Mitigating Activation Divergence for Accurate and Communication-Efficient Federated Learning
• Federated learning with class imbalance reduction
• Addressing Class Imbalance in Federated Learning
• Stochastic Distributed Optimization for Machine Learning from Decentralized Features