SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Ranks

Yue Xia, Tayyebeh Jahani-Nezhad, Mayank Bakshi +1

We consider federated parameter-efficient fine-tuning of foundation models, e.g., LLMs and LVMs, using low-rank adaptation (LoRA; ~Hu et al.\ 2022). Combining LoRA with federated l…