Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insights

Abstract

Federated learning has the potential to unlock siloed data and distributed resources by enabling collaborative model training without sharing private data. As more complex foundational models gain widespread use, the need to expand training resources and integrate privately owned data grows as well. In this article, we explore the intersection of federated learning and foundational models, aiming to identify, categorize, and characterize technical methods that integrate the two paradigms. As a unified survey is currently unavailable, we present a literature survey structured around a novel taxonomy that follows the development life-cycle stages, along with a technical comparison of available methods. Additionally, we provide practical insights and guidelines for implementing and evolving these methods, with a specific focus on the healthcare domain as a case study, where the potential impact of federated learning and foundational models is considered significant. Our survey covers multiple intersecting topics, including but not limited to federated learning, self-supervised learning, fine-tuning, distillation, and transfer learning. Initially, we retrieved and reviewed a set of over 4,200 articles. This collection was narrowed to more than 250 thoroughly reviewed articles through inclusion criteria, featuring 42 unique methods. The methods were used to construct the taxonomy and enabled their comparison based on complexity, efficiency, and scalability. We present these results as a self-contained overview that not only summarizes the state of the field but also provides insights into the practical aspects of adopting, evolving, and integrating foundational models with federated learning.

References in corpus (18)

• Bootstrap your own latent: A new approach to self-supervised Learning
• Continuous Integration, Delivery and Deployment: A Systematic Review on Approaches, Tools, Challenges and Practices
• QLoRA: Efficient Finetuning of Quantized LLMs
• Federated Learning from Pre-Trained Models: A Contrastive Learning Approach
• Foundational Models in Medical Imaging: A Comprehensive Survey and Future Vision
• Divergence-aware Federated Self-Supervised Learning
• SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models
• InfoNCE Loss Provably Learns Cluster-Preserving Representations
• FedTune: A Deep Dive into Efficient Federated Fine-Tuning with Pre-trained Transformers
• FLoRA: Enhancing Vision-Language Models with Parameter-Efficient Federated Learning
• Leveraging Foundation Models to Improve Lightweight Clients in Federated Learning
• Federated Multilingual Models for Medical Transcript Analysis
• FL-TAC: Enhanced Fine-Tuning in Federated Learning via Low-Rank, Task-Specific Adapter Clustering
• FEDMEKI: A Benchmark for Scaling Medical Foundation Models via Federated Knowledge Injection
• Bridging the Gap Between Foundation Models and Heterogeneous Federated Learning
• FedFN: Feature Normalization for Alleviating Data Heterogeneity Problem in Federated Learning
• Federated LoRA with Sparse Communication
• Federated Learning for Inference at Anytime and Anywhere