A Mechanism Study of Delayed Loss Spikes in Batch-Normalized Linear Models

Peifeng Gao, Wenyi Fang, Yang Zheng +1

Delayed loss spikes have been reported in neural-network training, but existing theory mainly explains earlier non-monotone behavior caused by overly large fixed learning rates. We…