Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insights

Abstract

Federated learning has the potential to unlock siloed data and distributed resources by enabling collaborative model training without sharing private data. As more complex foundational models gain widespread use, the need to expand training resources and integrate privately owned data grows as well. In this article, we explore the intersection of federated learning and foundational models, aiming to identify, categorize, and characterize technical methods that integrate the two paradigms. As a unified survey is currently unavailable, we present a literature survey structured around a novel taxonomy that follows the development life-cycle stages, along with a technical comparison of available methods. Additionally, we provide practical insights and guidelines for implementing and evolving these methods, with a specific focus on the healthcare domain as a case study, where the potential impact of federated learning and foundational models is considered significant. Our survey covers multiple intersecting topics, including but not limited to federated learning, self-supervised learning, fine-tuning, distillation, and transfer learning. Initially, we retrieved and reviewed a set of over 4,200 articles. This collection was narrowed to more than 250 thoroughly reviewed articles through inclusion criteria, featuring 42 unique methods. The methods were used to construct the taxonomy and enabled their comparison based on complexity, efficiency, and scalability. We present these results as a self-contained overview that not only summarizes the state of the field but also provides insights into the practical aspects of adopting, evolving, and integrating foundational models with federated learning.

References in corpus (50)

• Efficient Estimation of Word Representations in Vector Space
• A Simple Framework for Contrastive Learning of Visual Representations
• Knowledge Distillation: A Survey
• Bootstrap your own latent: A new approach to self-supervised Learning
• On the Opportunities and Risks of Foundation Models
• Continuous Integration, Delivery and Deployment: A Systematic Review on Approaches, Tools, Challenges and Practices
• FedKD: Communication Efficient Federated Learning via Knowledge Distillation
• QLoRA: Efficient Finetuning of Quantized LLMs
• VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning
• A Survey on Mixture of Experts in Large Language Models
• Emerging Trends in Federated Learning: From Model Fusion to Federated X Learning
• Federated Learning from Pre-Trained Models: A Contrastive Learning Approach
• Foundational Models in Medical Imaging: A Comprehensive Survey and Future Vision
• Divergence-aware Federated Self-Supervised Learning
• DENSE: Data-Free One-Shot Federated Learning
• When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directions
• On the duality between contrastive and non-contrastive self-supervised learning
• Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Language
• Integration of Large Language Models and Federated Learning
• Federated Self-supervised Learning for Heterogeneous Clients
• InfoNCE Loss Provably Learns Cluster-Preserving Representations
• SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models
• HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning
• FedTune: A Deep Dive into Efficient Federated Fine-Tuning with Pre-trained Transformers
• Advances and Open Challenges in Federated Foundation Models
• Grounding Foundation Models through Federated Transfer Learning: A General Framework
• pFedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuning
• FeDeRA:Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decomposition
• The Future of Large Language Model Pre-training is Federated
• Federated Domain-Specific Knowledge Transfer on Large Language Models Using Synthetic Data
• Federated Adaptive Prompt Tuning for Multi-Domain Collaborative Learning
• FLoRA: Enhancing Vision-Language Models with Parameter-Efficient Federated Learning
• Dual-Personalizing Adapter for Federated Foundation Models
• Leveraging Foundation Models to Improve Lightweight Clients in Federated Learning
• Guiding The Last Layer in Federated Learning with Pre-Trained Models
• Federated Word2Vec: Leveraging Federated Learning to Encourage Collaborative Representation Learning
• SPD-CFL: Stepwise Parameter Dropout for Efficient Continual Federated Learning
• FL-TAC: Enhanced Fine-Tuning in Federated Learning via Low-Rank, Task-Specific Adapter Clustering
• ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization
• Federated Multilingual Models for Medical Transcript Analysis
• Mutual Enhancement of Large and Small Language Models with Cross-Silo Knowledge Transfer
• Position Paper: Assessing Robustness, Privacy, and Fairness in Federated Learning Integrated with Foundation Models
• Federated Learning for Inference at Anytime and Anywhere
• FEDMEKI: A Benchmark for Scaling Medical Foundation Models via Federated Knowledge Injection
• Bridging the Gap Between Foundation Models and Heterogeneous Federated Learning
• FedFN: Feature Normalization for Alleviating Data Heterogeneity Problem in Federated Learning
• Federated LoRA with Sparse Communication
• Synergizing Foundation Models and Federated Learning: A Survey
• Towards Federated Low-Rank Adaptation of Language Models with Rank Heterogeneity
• FDAPT: Federated Domain-adaptive Pre-training for Language Models