Rotated Manifold Optimization for Low-Rank Adaptation

Yuhui Ding, Javier Zazo, James Hensman

We propose a novel optimizer for low-rank adaptation (LoRA) that explicitly incorporates the gauge symmetry of low-rank factorization. Our optimizer extends recent matrix optimizer…