No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data

Abstract

A central challenge in training classification models in the real-world federated system is learning with non-IID data. To cope with this, most of the existing works involve enforcing regularization in local optimization or improving the model aggregation scheme at the server. Other works also share public datasets or synthesized samples to supplement the training of under-represented classes or introduce a certain level of personalization. Though effective, they lack a deep understanding of how the data heterogeneity affects each layer of a deep classification model. In this paper, we bridge this gap by performing an experimental analysis of the representations learned by different layers. Our observations are surprising: (1) there exists a greater bias in the classifier than other layers, and (2) the classification performance can be significantly improved by post-calibrating the classifier after federated training. Motivated by the above findings, we propose a novel and simple algorithm called Classifier Calibration with Virtual Representations (CCVR), which adjusts the classifier using virtual representations sampled from an approximated gaussian mixture model. Experimental results demonstrate that CCVR achieves state-of-the-art performance on popular federated learning benchmarks including CIFAR-10, CIFAR-100, and CINIC-10. We hope that our simple yet effective method can shed some light on the future research of federated learning with non-IID data.

arXiv comment22 pages, NeurIPS 2021

References in corpus (24)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning with Non-IID Data
• On the Convergence of FedAvg on Non-IID Data
• Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
• Federated Learning with Personalization Layers
• Ensemble Distillation for Robust Model Fusion in Federated Learning
• Federated Optimization in Heterogeneous Networks
• Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data
• Similarity of Neural Network Representations Revisited
• Improving Federated Learning Personalization via Model Agnostic Meta Learning
• Personalized Federated Learning: A Meta-Learning Approach
• FedML: A Research Library and Benchmark for Federated Machine Learning
• Federated Meta-Learning with Fast Convergence and Efficient Communication
• Bayesian Nonparametric Federated Learning of Neural Networks
• Robust Federated Learning in a Heterogeneous Environment
• Think Locally, Act Globally: Federated Learning with Local and Global Representations
• Federated Learning Based on Dynamic Regularization
• Federated Learning with Matched Averaging
• The Non-IID Data Quagmire of Decentralized Machine Learning
• Federated Learning on Non-IID Data Silos: An Experimental Study
• No Peek: A Survey of private distributed deep learning
• Federated Learning via Synthetic Data
• Federated User Representation Learning
• Towards Fair Federated Learning with Zero-Shot Data Augmentation

Cited by in corpus (5)

• DualFed: Enjoying both Generalization and Personalization in Federated Learning via Hierachical Representations
• FLea: Addressing Data Scarcity and Label Skew in Federated Learning via Privacy-preserving Feature Augmentation
• FLea: Addressing Data Scarcity and Label Skew in Federated Learning via Privacy-preserving Feature Augmentation
• Towards Fast and Stable Federated Learning: Confronting Heterogeneity via Knowledge Anchor
• PIP: Prototypes-Injected Prompt for Federated Class Incremental Learning