GaLore$+$: Boosting Low-Rank Adaptation for LLMs with Cross-Head Projection

Xutao Liao, Shaohui Li, Yuhui Xu +3

Recent low-rank training methods, such as GaLore, have significantly reduced the memory required to optimize large language models (LLMs). However, these methods often suffer from…