Understanding Decoupled and Early Weight Decay

Johan Bjorck, Kilian Weinberger, Carla Gomes

Weight decay (WD) is a traditional regularization technique in deep learning, but despite its ubiquity, its behavior is still an area of active research. Golatkar et al. have recen…