Fed-ensemble: Improving Generalization through Model Ensembling in Federated Learning

Abstract

In this paper we propose Fed-ensemble: a simple approach that bringsmodel ensembling to federated learning (FL). Instead of aggregating localmodels to update a single global model, Fed-ensemble uses random permutations to update a group of K models and then obtains predictions through model averaging. Fed-ensemble can be readily utilized within established FL methods and does not impose a computational overhead as it only requires one of the K models to be sent to a client in each communication round. Theoretically, we show that predictions on newdata from all K models belong to the same predictive posterior distribution under a neural tangent kernel regime. This result in turn sheds light onthe generalization advantages of model averaging. We also illustrate thatFed-ensemble has an elegant Bayesian interpretation. Empirical results show that our model has superior performance over several FL algorithms,on a wide range of data sets, and excels in heterogeneous settings often encountered in FL applications.

References in corpus (23)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• The Future of Digital Health with Federated Learning
• Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edge
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Federated Learning for Mobile Keyboard Prediction
• Federated Machine Learning: Concept and Applications
• Federated Learning with Personalization Layers
• On Exact Computation with an Infinitely Wide Neural Net
• Improving Federated Learning Personalization via Model Agnostic Meta Learning
• Deep Ensembles: A Loss Landscape Perspective
• Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm
• Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
• Federated Learning of a Mixture of Global and Local Models
• Ditto: Fair and Robust Federated Learning Through Personalization
• Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
• Bayesian Model-Agnostic Meta-Learning
• Federated Evaluation of On-device Personalization
• Distributed Learning with Compressed Gradient Differences
• FedSplit: An algorithmic framework for fast federated optimization
• The Internet of Federated Things (IoFT): A Vision for the Future and In-depth Survey of Data-driven Approaches for Federated Learning
• Uncertainty in Gradient Boosting via Ensembles
• A Double Residual Compression Algorithm for Efficient Distributed Learning
• FL-NTK: A Neural Tangent Kernel-based Framework for Federated Learning Convergence Analysis

Cited by in corpus (1)

• Collaborative and Federated Black-box Optimization: A Bayesian Optimization Perspective