Two Stage Wireless Federated LoRA Fine-Tuning with Sparsified Orthogonal Updates

Bumjun Kim, Wan Choi

Federated fine-tuning with low-rank adaptation (LoRA) communicates only two low-rank matrices instead of the full model, but existing methods typically fix the LoRA rank in advance…