Nora: Normalized Orthogonal Row Alignment for Scalable Matrix Optimizer

Jinghui Yuan, Jiaxuan Zou, Shuo Wang +2

Matrix-based optimizers have demonstrated immense potential in training Large Language Models (LLMs), however, designing an ideal optimizer remains a formidable challenge. A superi…