Statistical Inference for Rank Allocation in Low-Rank Adaptation

Yihang Gao, Vincent Y. F. Tan

Low-rank adaptation (LoRA) has become a widely used parameter-efficient fine-tuning method for large language models. Since different modules and layers may contribute unequally to…