Norm matters: efficient and accurate normalization schemes in deep networks

Abstract

Over the past few years, Batch-Normalization has been commonly used in deep networks, allowing faster training and high performance for a wide variety of applications. However, the reasons behind its merits remained unanswered, with several shortcomings that hindered its use for certain tasks. In this work, we present a novel view on the purpose and function of normalization methods and weight-decay, as tools to decouple weights' norm from the underlying optimized objective. This property highlights the connection between practices such as normalization, weight decay and learning-rate adjustments. We suggest several alternatives to the widely used $L^2$ batch-norm, using normalization in $L^1$ and $L^{\infty }$ spaces that can substantially improve numerical stability in low-precision implementations as well as provide computational and memory benefits. We demonstrate that such methods enable the first batch-norm alternative to work for half-precision implementations. Finally, we suggest a modification to weight-normalization, which improves its performance on large-scale tasks.

arXiv commenthttp://papers.nips.cc/paper/7485-norm-matters-efficient-and-accurate-normalization-schemes-in-deep-networks

Cited by in corpus (49)

• RANet: Ranking Attention Network for Fast Video Object Segmentation
• Scalable Methods for 8-bit Training of Neural Networks
• Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks
• Review: Deep Learning in Electron Microscopy
• AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights
• GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training
• Evolving Normalization-Activation Layers
• PruneTrain: Fast Neural Network Training by Dynamic Sparse Model Reconfiguration
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• Three Mechanisms of Weight Decay Regularization
• Four Things Everyone Should Know to Improve Batch Normalization
• An Exponential Learning Rate Schedule for Deep Learning
• Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data
• On the training dynamics of deep networks with $L_2$ regularization
• Learned Threshold Pruning
• Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence
• Towards an Adversarially Robust Normalization Approach
• Shape Matters: Understanding the Implicit Bias of the Noise Covariance
• Quantum State Tomography with Conditional Generative Adversarial Networks
• Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics
• PruneNet: Channel Pruning via Global Importance
• Extended Batch Normalization
• Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Rate
• Batch Group Normalization
• Positional Normalization
• Regularizing activations in neural networks via distribution matching with the Wasserstein metric
• Deep Learning Supersampled Scanning Transmission Electron Microscopy
• Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence
• Adaptive Loss Scaling for Mixed Precision Training
• Spherical Motion Dynamics: Learning Dynamics of Neural Network with Normalization, Weight Decay, and SGD
• Adaptive L2 Regularization in Person Re-Identification
• Implicit Regularization and Convergence for Weight Normalization
• Effectiveness of Scaled Exponentially-Regularized Linear Units (SERLUs)
• Deep learning for pedestrians: backpropagation in CNNs
• Boosting CNN beyond Label in Inverse Problems
• Batch norm with entropic regularization turns deterministic autoencoders into generative models
• Optimal Quantization for Batch Normalization in Neural Network Deployments and Beyond
• On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decay
• Scalable and Practical Natural Gradient for Large-Scale Deep Learning
• Layer-wise Conditioning Analysis in Exploring the Learning Dynamics of DNNs
• Initialization and Regularization of Factorized Neural Layers
• Enabling Binary Neural Network Training on the Edge
• Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks
• Finet: Using Fine-grained Batch Normalization to Train Light-weight Neural Networks
• E2-Train: Training State-of-the-art CNNs with Over 80% Energy Savings
• Towards Fully 8-bit Integer Inference for the Transformer Model
• New Interpretations of Normalization Methods in Deep Learning
• An Investigation into the Stochasticity of Batch Whitening
• Group Whitening: Balancing Learning Efficiency and Representational Capacity