L2 Regularization versus Batch and Weight Normalization

Abstract

Batch Normalization is a commonly used trick to improve the training of deep neural networks. These neural networks use L2 regularization, also called weight decay, ostensibly to prevent overfitting. However, we show that L2 regularization has no regularizing effect when combined with normalization. Instead, regularization has an influence on the scale of weights, and thereby on the effective learning rate. We investigate this dependence, both in theory, and experimentally. We show that popular optimization methods such as ADAM only partially eliminate the influence of normalization on the learning rate. This leads to a discussion on other ways to mitigate this issue.

References in corpus (1)

• Adam: A Method for Stochastic Optimization

Cited by in corpus (52)

• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
• Fixup Initialization: Residual Learning Without Normalization
• Review: Deep Learning in Electron Microscopy
• Pruning neural networks without any data by iteratively conserving synaptic flow
• Real-time Plant Health Assessment Via Implementing Cloud-based Scalable Transfer Learning On AWS DeepLens
• Uses and Abuses of the Cross-Entropy Loss: Case Studies in Modern Deep Learning
• Batch Normalization is a Cause of Adversarial Vulnerability
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• Three Mechanisms of Weight Decay Regularization
• An Exponential Learning Rate Schedule for Deep Learning
• Four Things Everyone Should Know to Improve Batch Normalization
• Online Normalization for Training Neural Networks
• On the training dynamics of deep networks with $L_2$ regularization
• Learned Threshold Pruning
• Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence
• Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networks
• EvoJAX: Hardware-Accelerated Neuroevolution
• Improving Pre-trained Language Model Fine-tuning with Noise Stability Regularization
• Learning to be a Statistician: Learned Estimator for Number of Distinct Values
• How to decay your learning rate
• Regularization Matters: A Nonparametric Perspective on Overparametrized Neural Network
• Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model
• Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics
• Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspective
• A Machine Learning Approach to Integral Field Unit Spectroscopy Observations: II. HII Region LineRatios
• Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Rate
• Positional Normalization
• Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theory
• Batch Normalization Preconditioning for Neural Network Training
• A statistical theory of cold posteriors in deep neural networks
• On Infinite-Width Hypernetworks
• Adaptive L2 Regularization in Person Re-Identification
• Fast Certified Robust Training with Short Warmup
• Spherical Motion Dynamics: Learning Dynamics of Neural Network with Normalization, Weight Decay, and SGD
• Shredder: Learning Noise Distributions to Protect Inference Privacy
• Real-Time Damage Detection in Fiber Lifting Ropes Using Lightweight Convolutional Neural Networks
• Self-Orthogonality Module: A Network Architecture Plug-in for Learning Orthogonal Filters
• Gradient-Coherent Strong Regularization for Deep Neural Networks
• A Systematic Comparison of Deep Learning Architectures in an Autonomous Vehicle
• Scalable and Practical Natural Gradient for Large-Scale Deep Learning
• On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decay
• FixNorm: Dissecting Weight Decay for Training Deep Neural Networks
• SQWA: Stochastic Quantized Weight Averaging for Improving the Generalization Capability of Low-Precision Deep Neural Networks
• Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks
• Understanding the Disharmony between Weight Normalization Family and Weight Decay: $\epsilon -$shifted $L_2$ Regularizer
• Strategies for Language Identification in Code-Mixed Low Resource Languages
• Modulating Regularization Frequency for Efficient Compression-Aware Model Training
• Topologically Densified Distributions
• Efficient Modelling Across Time of Human Actions and Interactions
• Regularization by Misclassification in ReLU Neural Networks
• Towards Accurate Knowledge Transfer via Target-awareness Representation Disentanglement