ARO: A New Lens On Matrix Optimization For Large Models

Wenbo Gong, Javier Zazo, Qijun Luo +3

Matrix-based optimizers have attracted growing interest for improving LLM training efficiency, with significant progress centered on orthogonalization/whitening based methods. Whil…