Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initialization

Etienne Boursier, Matthew Bowditch, Matthias Englert +1

The optimization of neural networks under weight decay remains poorly understood from a theoretical standpoint. While weight decay is standard practice in modern training procedure…