Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks

Abstract

While the authors of Batch Normalization (BN) identify and address an important problem involved in training deep networks-- Internal Covariate Shift-- the current solution has certain drawbacks. Specifically, BN depends on batch statistics for layerwise input normalization during training which makes the estimates of mean and standard deviation of input (distribution) to hidden layers inaccurate for validation due to shifting parameter values (especially during initial training epochs). Also, BN cannot be used with batch-size 1 during training. We address these drawbacks by proposing a non-adaptive normalization technique for removing internal covariate shift, that we call Normalization Propagation. Our approach does not depend on batch statistics, but rather uses a data-independent parametric estimate of mean and standard-deviation in every layer thus being computationally faster compared with BN. We exploit the observation that the pre-activation before Rectified Linear Units follow Gaussian distribution in deep networks, and that once the first and second order statistics of any given dataset are normalized, we can forward propagate this normalization without the need for recalculating the approximate statistics for hidden layers.

11 pages, ICML 2016, appendix added to the last version

References in corpus (3)

• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Learning Activation Functions to Improve Deep Neural Networks
• Natural Neural Networks

Cited by in corpus (31)

• Spectral Normalization for Generative Adversarial Networks
• Regularized Deep Networks in Intelligent Transportation Systems: A Taxonomy and a Case Study
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• Revisiting Small Batch Training for Deep Neural Networks
• Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
• Thinking Fast and Slow with Deep Learning and Tree Search
• Sharp Minima Can Generalize For Deep Nets
• Micro-Batch Training with Batch-Channel Normalization and Weight Standardization
• Root Mean Square Layer Normalization
• On the Effects of Batch and Weight Normalization in Generative Adversarial Networks
• Optimization on Submanifolds of Convolution Kernels in CNNs
• A Comprehensive and Modularized Statistical Framework for Gradient Norm Equality in Deep Neural Networks
• Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization
• Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks
• Batch Kalman Normalization: Towards Training Deep Neural Networks with Micro-Batches
• The Benefits of Over-parameterization at Initialization in Deep ReLU Networks
• Extended Batch Normalization
• Reducing Bias in Production Speech Models
• Momentum^2 Teacher: Momentum Teacher with Momentum Statistics for Self-Supervised Learning
• Making EfficientNet More Efficient: Exploring Batch-Independent Normalization, Group Convolutions and Reduced Resolution Training
• Normalization of Neural Networks using Analytic Variance Propagation
• A Robust Initialization of Residual Blocks for Effective ResNet Training without Batch Normalization
• Channel Equilibrium Networks for Learning Deep Representation
• Mean Shift Rejection: Training Deep Neural Networks Without Minibatch Statistics or Normalization
• Feed-forward Uncertainty Propagation in Belief and Neural Networks
• Information Geometrically Generalized Covariate Shift Adaptation
• Is Batch Norm unique? An empirical investigation and prescription to emulate the best properties of common normalizers without batch dependence
• Accelerating Training of Deep Neural Networks with a Standardization Loss
• On Optimality Conditions for Auto-Encoder Signal Recovery
• Finet: Using Fine-grained Batch Normalization to Train Light-weight Neural Networks
• Double Forward Propagation for Memorized Batch Normalization