Retraction-Free Optimization over the Stiefel Manifold for the LoRA Fine-Tuning

Yuan Zhang, Jiang Hu, Zhijian Lai +2

Optimization over the Stiefel manifold plays a significant role in various machine learning tasks. Existing methods either use the retraction operators, requiring costly orthonorma…