Weight-norm Criticality: A Mechanism for Loss Spikes Induced by the Normalization and Weight Decay

Xiaolong Li, Zhangchen Zhou, Zhi-Qin John Xu

Most explanations of training instability focus on \emph{learning-rate criticality}, typically characterized by the Edge of Stability, beyond which optimization becomes unstable. W…