Federated Learning with Matched Averaging

Abstract

Federated learning allows edge devices to collaboratively learn a shared model while keeping the training data on device, decoupling the ability to do model training from the need to store the data in the cloud. We propose Federated matched averaging (FedMA) algorithm designed for federated learning of modern neural network architectures e.g. convolutional neural networks (CNNs) and LSTMs. FedMA constructs the shared global model in a layer-wise manner by matching and averaging hidden elements (i.e. channels for convolution layers; hidden states for LSTM; neurons for fully connected layers) with similar feature extraction signatures. Our experiments indicate that FedMA not only outperforms popular state-of-the-art federated learning algorithms on deep CNN and LSTM architectures trained on real world datasets, but also reduces the overall communication burden.

Accepted by ICLR 2020

References in corpus (7)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• On the Convergence of Adam and Beyond
• Towards Federated Learning at Scale: System Design
• LEAF: A Benchmark for Federated Settings
• No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing World
• Bayesian Nonparametric Federated Learning of Neural Networks

Cited by in corpus (48)

• Ensemble Distillation for Robust Model Fusion in Federated Learning
• From Distributed Machine Learning to Federated Learning: A Survey
• A Field Guide to Federated Optimization
• A Federated Learning Aggregation Algorithm for Pervasive Computing: Evaluation and Comparison
• Variational Federated Multi-Task Learning
• Robust Federated Learning with Noisy Labels
• Federated Multi-Task Learning under a Mixture of Distributions
• The Non-IID Data Quagmire of Decentralized Machine Learning
• Federated Learning on Non-IID Data Silos: An Experimental Study
• FedMix: Approximation of Mixup under Mean Augmented Federated Learning
• FedLab: A Flexible Federated Learning Framework
• FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout
• Towards Utilizing Unlabeled Data in Federated Learning: A Survey and Prospective
• No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data
• Auto-FedAvg: Learnable Federated Averaging for Multi-Institutional Medical Image Segmentation
• Fairness and Accuracy in Federated Learning
• Auto-weighted Robust Federated Learning with Corrupted Data Sources
• A distillation-based approach integrating continual learning and federated learning for pervasive services
• Federated Continual Learning with Weighted Inter-client Transfer
• Federated Learning Meets Multi-objective Optimization
• A Systematic Literature Review on Federated Learning: From A Model Quality Perspective
• Model-Contrastive Federated Learning
• Adversarial training in communication constrained federated learning
• FedCCEA : A Practical Approach of Client Contribution Evaluation for Federated Learning
• Personalized Federated Learning with Gaussian Processes
• Towards Unsupervised Domain Adaptation for Deep Face Recognition under Privacy Constraints via Federated Learning
• Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning
• Hybrid Federated Learning: Algorithms and Implementation
• Federated Learning on Non-IID Data: A Survey
• Model Fusion via Optimal Transport
• Edge Bias in Federated Learning and its Solution by Buffered Knowledge Distillation
• Decentralised Person Re-Identification with Selective Knowledge Aggregation
• Personalized Retrogress-Resilient Framework for Real-World Medical Federated Learning
• Federated Learning for Open Banking
• On Large-Cohort Training for Federated Learning
• Federated CycleGAN for Privacy-Preserving Image-to-Image Translation
• Critical Learning Periods in Federated Learning
• An Operator Splitting View of Federated Learning
• EasyFL: A Low-code Federated Learning Platform For Dummies
• Dynamic Attention-based Communication-Efficient Federated Learning
• Adversarial Robustness through Bias Variance Decomposition: A New Perspective for Federated Learning
• ResiliNet: Failure-Resilient Inference in Distributed Neural Networks
• Exact Support Recovery in Federated Regression with One-shot Communication
• FedNS: Improving Federated Learning for collaborative image classification on mobile clients
• Neural Tangent Kernel Empowered Federated Learning
• Fed2: Feature-Aligned Federated Learning
• Federated Learning from Small Datasets
• FLBench: A Benchmark Suite for Federated Learning