Three Mechanisms of Weight Decay Regularization

Abstract

Weight decay is one of the standard tricks in the neural network toolbox, but the reasons for its regularization effect are poorly understood, and recent results have cast doubt on the traditional interpretation in terms of $L_2$ regularization. Literal weight decay has been shown to outperform $L_2$ regularization for optimizers for which they differ. We empirically investigate weight decay for three optimization algorithms (SGD, Adam, and K-FAC) and a variety of network architectures. We identify three distinct mechanisms by which weight decay exerts a regularization effect, depending on the particular optimization algorithm and architecture: (1) increasing the effective learning rate, (2) approximately regularizing the input-output Jacobian norm, and (3) reducing the effective damping coefficient for second-order optimization. Our results provide insight into how to improve the regularization of neural networks.

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Decoupled Weight Decay Regularization
• Wide Residual Networks
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• Adding Gradient Noise Improves Learning for Very Deep Networks
• L2 Regularization versus Batch and Weight Normalization
• Fisher-Rao Metric, Geometry, and Complexity of Neural Networks
• Norm matters: efficient and accurate normalization schemes in deep networks
• Noisy Natural Gradient as Variational Inference
• A Coordinate-Free Construction of Scalable Natural Gradient

Cited by in corpus (25)

• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• MixMatch: A Holistic Approach to Semi-Supervised Learning
• ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring
• Rethinking the Hyperparameters for Fine-tuning
• Robust Learning with Jacobian Regularization
• Gradient Centralization: A New Optimization Technique for Deep Neural Networks
• On the training dynamics of deep networks with $L_2$ regularization
• Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence
• On the Optimal Weighted ${\ell }_2$ Regularization in Overparameterized Linear Regression
• Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics
• SimPLE: Similar Pseudo Label Exploitation for Semi-Supervised Classification
• When Does Preconditioning Help or Hurt Generalization?
• An Empirical Study of Large-Batch Stochastic Gradient Descent with Structured Covariance Noise
• Eigenvalue Corrected Noisy Natural Gradient
• Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theory
• ConvNets for Counting: Object Detection of Transient Phenomena in Steelpan Drums
• FixNorm: Dissecting Weight Decay for Training Deep Neural Networks
• Understanding the Disharmony between Weight Normalization Family and Weight Decay: $\epsilon -$shifted $L_2$ Regularizer
• Volumization as a Natural Generalization of Weight Decay
• Invexifying Regularization of Non-Linear Least-Squares Problems
• Whitening and second order optimization both make information in the dataset unusable during training, and can reduce or prevent generalization
• L-CO-Net: Learned Condensation-Optimization Network for Clinical Parameter Estimation from Cardiac Cine MRI
• Modulating Regularization Frequency for Efficient Compression-Aware Model Training
• New Interpretations of Normalization Methods in Deep Learning
• LocalNorm: Robust Image Classification through Dynamically Regularized Normalization