Does Weight Decay Enhance Training Stability?

Marius Saether, Amir Kolic, Tomaso Poggio +1

In modern deep learning, weight decay is often credited with "stabilizing" training dynamics, diverging from its classical role as a static regularization penalty. We investigate a…