Multi-Level Analyzation of Imbalance to Resolve Non-IID-Ness in Federated Learning

Abstract

Class imbalance is a common problem in deep learning that severely degrades performance. In federated learning (FL), it is a critical factor contributing to non-identically distributed data (non-IID). Building on several previous attempts, we define and analyze imbalance issues in FL at three levels: inter-case, inter-class, and inter-client. Inter-case imbalance addresses the imbalance in every single class; inter-class imbalance compares the number of data between different classes. Inter-client imbalance represents different skewness of local data between clients. Based on these concepts, we propose FedBB, which consists of two main components: (1) Positive Negative Balanced (PNB) loss function addresses the inter-case and inter-class imbalances in local training, enhancing generalization on highly skewed local client datasets. It optimizes both multi-label and multi-class classifications by assigning higher weights to minority cases or classes. (2) Client Balanced Reweighting (CBR) reweights clients based on inter-client imbalance during model aggregation, giving greater weight to models trained on less skewed datasets. Various experiments on X-ray and natural image datasets demonstrate that FedBB outperforms other algorithms in both performance and efficiency. Additionally, it requires limited statistical information, which is beneficial for privacy protection. Through ablation studies, we proved that PNB loss and CBR independently contribute to performance. As FedBB aims to build a global model that accurately classifies all classes, it can serve as a baseline for the generic and personalized FL.

27 pages, 5 figures, 13 tables. Accepted for publication in Neurocomputing (2025). Author Accepted Manuscript

References in corpus (13)

• The Future of Digital Health with Federated Learning
• Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
• Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization
• FedMD: Heterogenous Federated Learning via Model Distillation
• One-Shot Federated Learning
• The Devil is in the Tails: Fine-grained Classification in the Wild
• Deep Active Learning over the Long Tail
• On Bridging Generic and Personalized Federated Learning for Image Classification
• Git Re-Basin: Merging Models modulo Permutation Symmetries
• ISFL: Federated Learning for Non-i.i.d. Data with Local Importance Sampling
• GradAug: A New Regularization Method for Deep Neural Networks
• Joint Local Relational Augmentation and Global Nash Equilibrium for Federated Learning with Non-IID Data
• Handling Inter-class and Intra-class Imbalance in Class-imbalanced Learning