Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devices

Fahao Chen, Jie Wan, Peng Li +2

Federated fine-tuning of Mixture-of-Experts (MoE)-based large language models (LLMs) is challenging due to their massive computational requirements and the resource constraints of…