ORCA: The Annealed Spectral Conditioning Optimizer for Faster, Better LLM Training

Yuanshi Liu, Boyuan Jiang, Liang Hou +4

Modern LLM optimizers such as Muon often produce weight matrices with higher effective rank than Adam, yet further spectral control has delivered only modest gains. We identify a t…