Model-Contrastive Federated Learning

Abstract

Federated learning enables multiple parties to collaboratively train a machine learning model without communicating their local data. A key challenge in federated learning is to handle the heterogeneity of local data distribution across parties. Although many studies have been proposed to address this challenge, we find that they fail to achieve high performance in image datasets with deep learning models. In this paper, we propose MOON: model-contrastive federated learning. MOON is a simple and effective federated learning framework. The key idea of MOON is to utilize the similarity between model representations to correct the local training of individual parties, i.e., conducting contrastive learning in model-level. Our extensive experiments show that MOON significantly outperforms the other state-of-the-art federated learning algorithms on various image classification tasks.

Accepted by CVPR 2021

References in corpus (18)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• A Simple Framework for Contrastive Learning of Visual Representations
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Bootstrap your own latent: A new approach to self-supervised Learning
• Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
• Contrastive Multiview Coding
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• FedML: A Research Library and Benchmark for Federated Machine Learning
• LEAF: A Benchmark for Federated Settings
• Personalized Federated Learning with Moreau Envelopes
• Bayesian Nonparametric Federated Learning of Neural Networks
• Federated Learning Based on Dynamic Regularization
• Federated Learning with Matched Averaging
• Federated Learning on Non-IID Data Silos: An Experimental Study
• Blockchain-Federated-Learning and Deep Learning Models for COVID-19 detection using CT Imaging
• Lower Bounds and Optimal Algorithms for Personalized Federated Learning
• The OARF Benchmark Suite: Characterization and Implications for Federated Learning Systems
• Personalized Cross-Silo Federated Learning on Non-IID Data