Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classification

Abstract

Batch normalization (BN) has become a de facto standard for training deep convolutional networks. However, BN accounts for a significant fraction of training run-time and is difficult to accelerate, since it is a memory-bandwidth bounded operation. Such a drawback of BN motivates us to explore recently proposed weight normalization algorithms (WN algorithms), i.e. weight normalization, normalization propagation and weight normalization with translated ReLU. These algorithms don't slow-down training iterations and were experimentally shown to outperform BN on relatively small networks and datasets. However, it is not clear if these algorithms could replace BN in practical, large-scale applications. We answer this question by providing a detailed comparison of BN and WN algorithms using ResNet-50 network trained on ImageNet. We found that although WN achieves better training accuracy, the final test accuracy is significantly lower ($\approx 6\%$) than that of BN. This result demonstrates the surprising strength of the BN regularization effect which we were unable to compensate for using standard regularization techniques like dropout and weight decay. We also found that training of deep networks with WN algorithms is significantly less stable compared to BN, limiting their practical applications.

References in corpus (11)

• Deep Residual Learning for Image Recognition
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Striving for Simplicity: The All Convolutional Net
• Wide Residual Networks
• Improved Techniques for Training GANs
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• Aggregated Residual Transformations for Deep Neural Networks
• Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
• All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulation
• Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning

Cited by in corpus (20)

• High-Performance Large-Scale Image Recognition Without Normalization
• Micro-Batch Training with Batch-Channel Normalization and Weight Standardization
• Review: Deep Learning in Electron Microscopy
• A Mean Field Theory of Batch Normalization
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• Online Normalization for Training Neural Networks
• A Comprehensive and Modularized Statistical Framework for Gradient Norm Equality in Deep Neural Networks
• A Simple Convergence Proof of Adam and Adagrad
• Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization
• Understanding and Resolving Performance Degradation in Graph Convolutional Networks
• SparseTrain:Leveraging Dynamic Sparsity in Training DNNs on General-Purpose SIMD Processors
• Optimization Theory for ReLU Neural Networks Trained with Normalization Layers
• Normalization of Neural Networks using Analytic Variance Propagation
• Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence
• Deep learning for pedestrians: backpropagation in CNNs
• Mean Shift Rejection: Training Deep Neural Networks Without Minibatch Statistics or Normalization
• Accelerating Training of Deep Neural Networks with a Standardization Loss
• "BNN - BN = ?": Training Binary Neural Networks without Batch Normalization
• Understanding the Disharmony between Weight Normalization Family and Weight Decay: $\epsilon -$shifted $L_2$ Regularizer
• Efficient Modelling Across Time of Human Actions and Interactions