Towards Federated Low-Rank Adaptation of Language Models with Rank Heterogeneity

Yuji Byun, Jaeho Lee

Low-rank adaptation (LoRA) offers an efficient alternative to full-weight adaptation in federated fine-tuning of language models, significantly reducing computational costs. By adj…