Memory-Efficient LLM Training by Various-Grained Low-Rank Projection of Gradients

Yezhen Wang, Zhouhao Yang, Brian K Chen +4

Building upon the success of low-rank adapter (LoRA), low-rank gradient projection (LoRP) has emerged as a promising solution for memory-efficient fine-tuning. However, existing Lo…