Understanding the Disharmony between Weight Normalization Family and Weight Decay: shifted Regularizer
arXiv:1911.05920
Abstract
The merits of fast convergence and potentially better performance of the weight normalization family have drawn increasing attention in recent years. These methods use standardization or normalization that changes the weight to , which makes independent to the magnitude of . Surprisingly, must be decayed during gradient descent, otherwise we will observe a severe under-fitting problem, which is very counter-intuitive since weight decay is widely known to prevent deep networks from over-fitting. In this paper, we \emph{theoretically} prove that the weight decay term merely modulates the effective learning rate for improving objective optimization, and has no influence on generalization when the weight normalization family is compositely employed. Furthermore, we also expose several critical problems when introducing weight decay term to weight normalization family, including the missing of global minimum and training instability. To address these problems, we propose an shifted regularizer, which shifts the objective by a positive constant . Such a simple operation can theoretically guarantee the existence of global minimum, while preventing the network weights from being too small and thus avoiding gradient float overflow. It significantly improves the training stability and can achieve slightly better performance in our practice. The effectiveness of shifted regularizer is comprehensively validated on the ImageNet, CIFAR-100, and COCO datasets. Our codes and pretrained models will be released in https://github.com/implus/PytorchInsight.
arXiv comment12 pages, 9 figures
References in corpus (21)
- Adam: A Method for Stochastic Optimization
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Decoupled Weight Decay Regularization
- An overview of gradient descent optimization algorithms
- mixup: Beyond Empirical Risk Minimization
- Instance Normalization: The Missing Ingredient for Fast Stylization
- Improved Regularization of Convolutional Neural Networks with Cutout
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- MobileNetV2: Inverted Residuals and Linear Bottlenecks
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
- MMDetection: Open MMLab Detection Toolbox and Benchmark
- Large Batch Training of Convolutional Networks
- Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
- L2 Regularization versus Batch and Weight Normalization
- Spatial Group-wise Enhance: Improving Semantic Feature Learning in Convolutional Networks
- Differentiable Learning-to-Normalize via Switchable Normalization
- Micro-Batch Training with Batch-Channel Normalization and Weight Standardization
- On the Effects of Batch and Weight Normalization in Generative Adversarial Networks
- Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classification
- Three Mechanisms of Weight Decay Regularization