Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors

Alexander Hägele, Alejandro Hernández-Cano, Atli Kosson +1

Modern neural network training relies on optimizers such as Adam and Muon which act on each weight matrix as a single object. Yet every weight matrix carries two distinct quantitie…