Between Gradient and Natural Gradient: A Continuum of LoRA Initializations

Dianze Liu, Farshid Ghezelbash

Low-rank adaptation (LoRA) fine-tunes large pretrained models at a fraction of the cost of full fine-tuning, but its performance depends strongly on how the adapters are initialize…