Ensemble Distillation for Robust Model Fusion in Federated Learning

Abstract

Federated Learning (FL) is a machine learning setting where many devices collaboratively train a machine learning model while keeping the training data decentralized. In most of the current training schemes the central model is refined by averaging the parameters of the server model and the updated parameters from the client side. However, directly averaging model parameters is only possible if all models have the same structure and size, which could be a restrictive constraint in many scenarios. In this work we investigate more powerful and more flexible aggregation schemes for FL. Specifically, we propose ensemble distillation for model fusion, i.e. training the central classifier through unlabeled data on the outputs of the models from the clients. This knowledge distillation technique mitigates privacy risk and cost to the same extent as the baseline FL algorithms, but allows flexible aggregation over heterogeneous client models that can differ e.g. in size, numerical precision or structure. We show in extensive empirical experiments on various CV/NLP datasets (CIFAR-10/100, ImageNet, AG News, SST2) and settings (heterogeneous models/data) that the server model can be trained much faster, requiring fewer communication rounds than any existing FL technique so far.

References in corpus (37)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Towards Federated Learning at Scale: System Design
• Binarized Neural Networks
• SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
• Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
• Differentially Private Federated Learning: A Client Level Perspective
• FedMD: Heterogenous Federated Learning via Model Distillation
• Federated Optimization in Heterogeneous Networks
• Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data
• Born Again Neural Networks
• A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
• Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
• Adaptive Personalized Federated Learning
• LEAF: A Benchmark for Federated Settings
• Agnostic Federated Learning
• Don't Use Large Mini-Batches, Use Local SGD
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• Large scale distributed neural network training through online distillation
• Overcoming Forgetting in Federated Learning on Non-IID Data
• One-Shot Federated Learning
• Adaptive Federated Optimization
• Algorithms and Theory for Multiple-Source Adaptation
• Federated Learning with Matched Averaging
• Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer
• The Non-IID Data Quagmire of Decentralized Machine Learning
• Dynamic Model Pruning with Feedback
• Distilled One-Shot Federated Learning
• Contrastive Representation Distillation
• FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning
• Federated Model Distillation with Noise-Free Differential Privacy
• FEED: Feature-level Ensemble for Knowledge Distillation

Cited by in corpus (71)

• Knowledge Distillation: A Survey
• A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection
• FedKD: Communication Efficient Federated Learning via Knowledge Distillation
• FedALA: Adaptive Local Aggregation for Personalized Federated Learning
• Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge
• Label-Efficient Self-Supervised Federated Learning for Tackling Data Heterogeneity in Medical Imaging
• FedCP: Separating Feature Information for Personalized Federated Learning via Conditional Policy
• No One Left Behind: Inclusive Federated Learning over Heterogeneous Devices
• An Experimental Study of Byzantine-Robust Aggregation Schemes in Federated Learning
• Federated Learning on Non-IID Data Silos: An Experimental Study
• Artificial Intelligence of Things: A Survey
• FedCG: Leverage Conditional GAN for Protecting Privacy and Maintaining Competitive Performance in Federated Learning
• Federated Learning Meets Natural Language Processing: A Survey
• No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data
• One-shot Federated Learning without Server-side Training
• Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data
• AnycostFL: Efficient On-Demand Federated Learning over Heterogeneous Edge Devices
• FedACK: Federated Adversarial Contrastive Knowledge Distillation for Cross-Lingual and Cross-Model Social Bot Detection
• FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning
• Is Aggregation the Only Choice? Federated Learning via Layer-wise Model Recombination
• Federated Active Learning (F-AL): an Efficient Annotation Strategy for Federated Learning
• FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusion
• Decentralized and Model-Free Federated Learning: Consensus-Based Distillation in Function Space
• AdaptiveFL: Adaptive Heterogeneous Federated Learning for Resource-Constrained AIoT Systems
• Uncertainty Minimization for Personalized Federated Semi-Supervised Learning
• Federated brain tumor segmentation: an extensive benchmark
• Edge-cloud Collaborative Learning with Federated and Centralized Features
• Personalized Federated Learning through Local Memorization
• Specialized federated learning using a mixture of experts
• FedBoosting: Federated Learning with Gradient Protected Boosting for Text Recognition
• Communication-Efficient Federated Distillation
• Federated Stain Normalization for Computational Pathology
• ProgFed: Effective, Communication, and Computation Efficient Federated Learning by Progressive Training
• Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learning
• What Do We Mean by Generalization in Federated Learning?
• From Federated Learning to Federated Neural Architecture Search: A Survey
• Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspective
• Personalized Federated Learning with Multi-branch Architecture
• Federated Learning on Non-iid Data via Local and Global Distillation
• Practical One-Shot Federated Learning for Cross-Silo Setting
• QuPeD: Quantized Personalization via Distillation with Applications to Federated Learning
• Edge Bias in Federated Learning and its Solution by Buffered Knowledge Distillation
• Distributed Dynamic Map Fusion via Federated Learning for Intelligent Networked Vehicles
• CrowdTransfer: Enabling Crowd Knowledge Transfer in AIoT Community
• LIA: Privacy-Preserving Data Quality Evaluation in Federated Learning Using a Lazy Influence Approximation
• Synthetic data shuffling accelerates the convergence of federated learning under data heterogeneity
• Towards Model Agnostic Federated Learning Using Knowledge Distillation
• Source Inference Attacks in Federated Learning
• Decentralised Person Re-Identification with Selective Knowledge Aggregation
• Ensemble Federated Adversarial Training with Non-IID data
• Communication Efficient Federated Learning with Energy Awareness over Wireless Networks
• Learning while Respecting Privacy and Robustness to Distributional Uncertainties and Adversarial Data
• Compare Where It Matters: Using Layer-Wise Regularization To Improve Federated Learning on Heterogeneous Data
• FedSC: Federated Learning with Semantic-Aware Collaboration
• Real-Time Decentralized knowledge Transfer at the Edge
• FedGPO: Heterogeneity-Aware Global Parameter Optimization for Efficient Federated Learning
• EasyFL: A Low-code Federated Learning Platform For Dummies
• Reward-Based 1-bit Compressed Federated Distillation on Blockchain
• Inclusive Data Representation in Federated Learning: A Novel Approach Integrating Textual and Visual Prompt
• Adaptive Distillation: Aggregating Knowledge from Multiple Paths for Efficient Distillation
• Federated Learning via Plurality Vote
• Vicinal Feature Statistics Augmentation for Federated 3D Medical Volume Segmentation
• Collaborative Semantic Aggregation and Calibration for Federated Domain Generalization
• FedRAD: Federated Robust Adaptive Distillation
• BOBA: Byzantine-Robust Federated Learning with Label Skewness
• Efficient Federated Learning for AIoT Applications Using Knowledge Distillation
• Masked Training of Neural Networks with Partial Gradients
• A Unified Solution to Diverse Heterogeneities in One-shot Federated Learning
• Weight Divergence Driven Divide-and-Conquer Approach for Optimal Federated Learning from non-IID Data
• Moss: Proxy Model-based Full-Weight Aggregation in Federated Learning with Heterogeneous Models
• Towards Explainable Multi-Party Learning: A Contrastive Knowledge Sharing Framework