On the Convergence of Stochastic Low-Rank Adaptation

Ru Wang, Chengchang Liu, John C. S. Lui

Low-rank adaptation (LoRA) optimizes $J(B,A)=\mathcal{L}(W_{\mathrm{base}}+sBA)$ over two adapters $B\in {\mathbb{R}}^{m\times r}$ and $A\in {\mathbb{R}}^{r\times n}$ that form a low-…