Pro-KLShampoo: Projected KL-Shampoo with Whitening Recovered by Orthogonalization

Ruotong Sun, Ermin Wei

Optimizers that exploit the matrix structure of gradients are central to modern LLM pre-training, with two distinct frontiers: explicit Kronecker-factored preconditioning -- most r…