GaLore 2: Large-Scale LLM Pre-Training by Gradient Low-Rank Projection

DiJia Su, Andrew Gu, Jane Xu +2

Large language models (LLMs) have revolutionized natural language understanding and generation but face significant memory bottlenecks during training. GaLore, Gradient Low-Rank Pr…