HD-PiSSA: High-Rank Distributed Orthogonal Adaptation

Yiding Wang, Fauxu Meng, Xuefeng Zhang +3

Existing parameter-efficient fine-tuning (PEFT) methods for large language models (LLMs), such as LoRA and PiSSA, constrain model updates to low-rank subspaces, limiting their expr…