No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Training

Noel Thomas

Memory-efficient optimizers such as GaLore train large language models by projecting gradients onto a rank-r subspace recomputed every T steps, assuming this subspace is a slowly d…