FISMO: Fisher-Structured Momentum-Orthogonalized Optimizer

Chenrui Xu, Wenjing Yan, Ying-Jun Angela Zhang

Training large-scale neural networks requires solving nonconvex optimization where the choice of optimizer fundamentally determines both convergence behavior and computational effi…