Variational Federated Multi-Task Learning

Abstract

In federated learning, a central server coordinates the training of a single model on a massively distributed network of devices. This setting can be naturally extended to a multi-task learning framework, to handle real-world federated datasets that typically show strong statistical heterogeneity among devices. Despite federated multi-task learning being shown to be an effective paradigm for real-world datasets, it has been applied only on convex models. In this work, we introduce VIRTUAL, an algorithm for federated multi-task learning for general non-convex models. In VIRTUAL the federated network of the server and the clients is treated as a star-shaped Bayesian network, and learning is performed on the network using approximated variational inference. We show that this method is effective on real-world federated datasets, outperforming the current state-of-the-art for federated learning, and concurrently allowing sparser gradient updates.

References in corpus (30)

• Distilling the Knowledge in a Neural Network
• Overcoming catastrophic forgetting in neural networks
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• An Overview of Multi-Task Learning in Deep Neural Networks
• Federated Learning with Non-IID Data
• Federated Optimization: Distributed Machine Learning for On-Device Intelligence
• Expectation Propagation for approximate Bayesian inference
• Continual Learning Through Synaptic Intelligence
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• EMNIST: an extension of MNIST to handwritten letters
• Applied Federated Learning: Improving Google Keyboard Query Suggestions
• Variational Dropout and the Local Reparameterization Trick
• Federated Optimization in Heterogeneous Networks
• Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data
• Progress & Compress: A scalable framework for continual learning
• Variational Continual Learning
• Overcoming Catastrophic Forgetting by Incremental Moment Matching
• LEAF: A Benchmark for Federated Settings
• Federated Collaborative Filtering for Privacy-Preserving Personalized Recommendation System
• Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches
• Word Representations via Gaussian Embedding
• Bayesian Nonparametric Federated Learning of Neural Networks
• Bayesian Recurrent Neural Networks
• Parallel training of DNNs with Natural Gradient and Parameter Averaging
• Streaming Variational Bayes
• Communication-Efficient Distributed Dual Coordinate Ascent
• Federated Learning with Matched Averaging
• Memory Replay GANs: learning to generate images from new categories without forgetting
• Partitioned Variational Inference: A unified framework encompassing federated and continual learning
• Scalable and Order-robust Continual Learning with Additive Parameter Decomposition

Cited by in corpus (36)

• A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection
• FedMD: Heterogenous Federated Learning via Model Distillation
• Three Approaches for Personalization with Applications to Federated Learning
• Federated Learning of a Mixture of Global and Local Models
• Federated Continual Learning via Knowledge Fusion: A Survey
• Privacy-preserving Artificial Intelligence Techniques in Biomedicine
• Federated Multi-Task Learning under a Mixture of Distributions
• Federated Learning for Healthcare Informatics
• Lower Bounds and Optimal Algorithms for Personalized Federated Learning
• A Bayesian Federated Learning Framework with Online Laplace Approximation
• Time-sensitive Learning for Heterogeneous Federated Edge Intelligence
• Fairness and Accuracy in Federated Learning
• Fed-Focal Loss for imbalanced data classification in Federated Learning
• Towards Flexible Device Participation in Federated Learning
• Graph Federated Learning for CIoT Devices in Smart Home Applications
• Client Selection for Federated Bayesian Learning
• Personalized Federated Learning for Intelligent IoT Applications: A Cloud-Edge based Framework
• A Systematic Literature Review on Federated Learning: From A Model Quality Perspective
• Dynamic Fusion based Federated Learning for COVID-19 Detection
• Federated Split Vision Transformer for COVID-19 CXR Diagnosis using Task-Agnostic Training
• Implicit Model Specialization through DAG-based Decentralized Federated Learning
• FedV: Privacy-Preserving Federated Learning over Vertically Partitioned Data
• Client Selection and Bandwidth Allocation in Wireless Federated Learning Networks: A Long-Term Perspective
• Personalized Federated Learning with Gaussian Processes
• Federated Learning on Non-IID Data: A Survey
• Towards Fair Federated Learning with Zero-Shot Data Augmentation
• SoK: Training Machine Learning Models over Multiple Sources with Privacy Preservation
• IFedAvg: Interpretable Data-Interoperability for Federated Learning
• Failure Prediction in Production Line Based on Federated Learning: An Empirical Study
• Channel-Driven Monte Carlo Sampling for Bayesian Distributed Learning in Wireless Data Centers
• QLSD: Quantised Langevin stochastic dynamics for Bayesian federated learning
• Real-Time Decentralized knowledge Transfer at the Edge
• Demystifying the Effects of Non-Independence in Federated Learning
• An Expectation-Maximization Perspective on Federated Learning
• Multiple Classification with Split Learning
• Optimization for Supervised Machine Learning: Randomized Algorithms for Data and Parameters