ROOT: Robust Orthogonalized Optimizer for Neural Network Training

Wei He, Kai Han, Hang Zhou +4

The optimization of large language models (LLMs) remains a critical challenge, particularly as model scaling exacerbates sensitivity to algorithmic imprecision and training instabi…