Flexible Clustered Federated Learning for Client-Level Data Distribution Shift

Abstract

Federated Learning (FL) enables the multiple participating devices to collaboratively contribute to a global neural network model while keeping the training data locally. Unlike the centralized training setting, the non-IID, imbalanced (statistical heterogeneity) and distribution shifted training data of FL is distributed in the federated network, which will increase the divergences between the local models and the global model, further degrading performance. In this paper, we propose a flexible clustered federated learning (CFL) framework named FlexCFL, in which we 1) group the training of clients based on the similarities between the clients' optimization directions for lower training divergence; 2) implement an efficient newcomer device cold start mechanism for framework scalability and practicality; 3) flexibly migrate clients to meet the challenge of client-level data distribution shift. FlexCFL can achieve improvements by dividing joint optimization into groups of sub-optimization and can strike a balance between accuracy and communication efficiency in the distribution shift environment. The convergence and complexity are analyzed to demonstrate the efficiency of FlexCFL. We also evaluate FlexCFL on several open datasets and made comparisons with related CFL frameworks. The results show that FlexCFL can significantly improve absolute test accuracy by +10.6% on FEMNIST compared to FedAvg, +3.5% on FashionMNIST compared to FedProx, +8.4% on MNIST compared to FeSEM. The experiment results show that FlexCFL is also communication efficient in the distribution shift environment.

arXiv commentManuscript under review. arXiv admin note: substantial text overlap with arXiv:2010.06870

References in corpus (15)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• TensorFlow: A system for large-scale machine learning
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Challenges, Methods, and Future Directions
• Federated Learning: Strategies for Improving Communication Efficiency
• Federated Learning with Non-IID Data
• A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection
• On the Convergence of FedAvg on Non-IID Data
• Communication Efficient Distributed Optimization using an Approximate Newton-type Method
• Agnostic Federated Learning
• Ditto: Fair and Robust Federated Learning Through Personalization
• Astraea: Self-balancing Federated Learning for Improving Classification Accuracy of Mobile Deep Learning Applications
• On perfect clustering of high dimension, low sample size data
• TornadoAggregate: Accurate and Scalable Federated Learning via the Ring-Based Architecture

Cited by in corpus (3)

• Towards Effective Clustered Federated Learning: A Peer-to-peer Framework with Adaptive Neighbor Matching
• Auxo: Efficient Federated Learning via Scalable Client Clustering
• Lazy But Effective: Collaborative Personalized Federated Learning with Heterogeneous Data