11 papers
Preventing Rank Collapse in Federated Low-Rank Adaptation with Client Heterogeneity
Fei Wu, Jia Hu, Geyong Min +1
Federated low-rank adaptation (FedLoRA) has facilitated communication-efficient and privacy-preserving fine-tuning of foundation models for downstream tasks. In practical federated…
BoHA: Blockwise Hadamard Product Adaptation for Parameter-Efficient Fine-Tuning
Feng Yu, Jia Hu, Geyong Min
Parameter-efficient fine-tuning (PEFT) of large language models trains a small task-specific parameter set while keeping the pretrained model frozen. The dominant Low-Rank Adaptati…
Task-agnostic Low-rank Residual Adaptation for Efficient Federated Continual Fine-Tuning
Feng Yu, Jia Hu, Geyong Min
Federated Parameter-Efficient Fine-Tuning (Fed-PEFT) enables lightweight adaptation of large pre-trained models in federated learning settings by updating only a small subset of pa…
Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptation
Fei Wu, Jia Hu, Geyong Min +1
Driven by the rapid growth of model parameters, parameter-efficient fine-tuning (PEFT) has become essential for adapting large models to diverse downstream tasks under constrained…
Adaptive Rank Allocation for Federated Parameter-Efficient Fine-Tuning of Language Models
Fei Wu, Jia Hu, Geyong Min +1
Pre-trained Language Models (PLMs) have demonstrated their superiority and versatility in modern Natural Language Processing (NLP), effectively adapting to various downstream tasks…
DeepFusion: Accelerating MoE Training via Federated Knowledge Distillation from Heterogeneous Edge Devices
Songyuan Li, Jia Hu, Ahmed M. Abdelmoniem +3
Recent Mixture-of-Experts (MoE)-based large language models (LLMs) such as Qwen-MoE and DeepSeek-MoE are transforming generative AI in natural language processing. However, these m…