L2 Regularization versus Batch and Weight Normalization

Abstract

Batch Normalization is a commonly used trick to improve the training of deep neural networks. These neural networks use L2 regularization, also called weight decay, ostensibly to prevent overfitting. However, we show that L2 regularization has no regularizing effect when combined with normalization. Instead, regularization has an influence on the scale of weights, and thereby on the effective learning rate. We investigate this dependence, both in theory, and experimentally. We show that popular optimization methods such as ADAM only partially eliminate the influence of normalization on the learning rate. This leads to a discussion on other ways to mitigate this issue.

Cited by in corpus (19)

• Fixup Initialization: Residual Learning Without Normalization
• Uses and Abuses of the Cross-Entropy Loss: Case Studies in Modern Deep Learning
• Batch Normalization is a Cause of Adversarial Vulnerability
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• An Exponential Learning Rate Schedule for Deep Learning
• Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence
• EvoJAX: Hardware-Accelerated Neuroevolution
• Learning to be a Statistician: Learned Estimator for Number of Distinct Values
• How to decay your learning rate
• Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics
• A Machine Learning Approach to Integral Field Unit Spectroscopy Observations: II. HII Region LineRatios
• Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Rate
• Self-Orthogonality Module: A Network Architecture Plug-in for Learning Orthogonal Filters
• FixNorm: Dissecting Weight Decay for Training Deep Neural Networks
• Scalable and Practical Natural Gradient for Large-Scale Deep Learning
• Understanding the Disharmony between Weight Normalization Family and Weight Decay: $\epsilon -$shifted $L_2$ Regularizer
• SQWA: Stochastic Quantized Weight Averaging for Improving the Generalization Capability of Low-Precision Deep Neural Networks
• Efficient Modelling Across Time of Human Actions and Interactions
• Regularization by Misclassification in ReLU Neural Networks