Joint Local Relational Augmentation and Global Nash Equilibrium for Federated Learning with Non-IID Data

Abstract

Federated learning (FL) is a distributed machine learning paradigm that needs collaboration between a server and a series of clients with decentralized data. To make FL effective in real-world applications, existing work devotes to improving the modeling of decentralized data with non-independent and identical distributions (non-IID). In non-IID settings, there are intra-client inconsistency that comes from the imbalanced data modeling, and inter-client inconsistency among heterogeneous client distributions, which not only hinders sufficient representation of the minority data, but also brings discrepant model deviations. However, previous work overlooks to tackle the above two coupling inconsistencies together. In this work, we propose FedRANE, which consists of two main modules, i.e., local relational augmentation (LRA) and global Nash equilibrium (GNE), to resolve intra- and inter-client inconsistency simultaneously. Specifically, in each client, LRA mines the similarity relations among different data samples and enhances the minority sample representations with their neighbors using attentive message passing. In server, GNE reaches an agreement among inconsistent and discrepant model deviations from clients to server, which encourages the global model to update in the direction of global optimum without breaking down the clients optimization toward their local optimums. We conduct extensive experiments on four benchmark datasets to show the superiority of FedRANE in enhancing the performance of FL with non-IID data.

arXiv commentTo appear in ACM International Conference on Multimedia (ACM MM23)

References in corpus (22)

• A Simple Framework for Contrastive Learning of Visual Representations
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• An Overview of Multi-Task Learning in Deep Neural Networks
• On the Convergence of FedAvg on Non-IID Data
• Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
• Federated Learning with Personalization Layers
• Stochastic gradient descent on Riemannian manifolds
• HybridAlpha: An Efficient Approach for Privacy-Preserving Federated Learning
• Ditto: Fair and Robust Federated Learning Through Personalization
• Personalized Federated Learning with Moreau Envelopes
• Variance Reduced Local SGD with Lower Communication Complexity
• SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum
• On Bridging Generic and Personalized Federated Learning for Image Classification
• PKD: General Distillation Framework for Object Detectors via Pearson Correlation Coefficient
• Conflict-Averse Gradient Descent for Multi-task Learning
• Federated Learning Meets Multi-objective Optimization
• Multi-Task Learning as a Bargaining Game
• Local Adaptivity in Federated Learning: Convergence and Consistency
• Addressing Algorithmic Disparity and Performance Inconsistency in Federated Learning
• Disentangled Federated Learning for Tackling Attributes Skew via Invariant Aggregation and Diversity Transferring
• Adapt to Adaptation: Learning Personalization for Cross-Silo Federated Learning
• PerAda: Parameter-Efficient Federated Learning Personalization with Generalization Guarantees

Cited by in corpus (1)

• Multi-Level Analyzation of Imbalance to Resolve Non-IID-Ness in Federated Learning