High-Dimensional Theory of LoRA Fine-Tuning in a Solvable Attention Model

O. Duranthon, F. Boncoraglio, L. Zdeborová

We develop a high-dimensional statistical theory of low-rank adaptation (LoRA) in attention models, capturing the interplay between pre-training and fine-tuning. We introduce a sol…