Loss Spike in Training Neural Networks

Xiaolong Li, Zhi-Qin John Xu, Zhongwang Zhang

In this work, we investigate the mechanism underlying loss spikes observed during neural network training. When the training enters a region with a lower-loss-as-sharper (LLAS) str…