OrScale: Orthogonalised Optimization with Layer-Wise Trust-Ratio Scaling

Yuxuan Lou, Yang You

Muon improves neural-network training by orthogonalizing matrix-valued updates, but it leaves each layer's update magnitude controlled mostly by a global learning rate. We introduc…