Normalization of Neural Networks using Analytic Variance Propagation

Abstract

We address the problem of estimating statistics of hidden units in a neural network using a method of analytic moment propagation. These statistics are useful for approximate whitening of the inputs in front of saturating non-linearities such as a sigmoid function. This is important for initialization of training and for reducing the accumulated scale and bias dependencies (compensating covariate shift), which presumably eases the learning. In batch normalization, which is currently a very widely applied technique, sample estimates of statistics of hidden units over a batch are used. The proposed estimation uses an analytic propagation of mean and variance of the training set through the network. The result depends on the network structure and its current weights but not on the specific batch input. The estimates are suitable for initialization and normalization, efficient to compute and independent of the batch size. The experimental verification well supports these claims. However, the method does not share the generalization properties of BN, to which our experiments give some additional insight.

References in corpus (11)

• Instance Normalization: The Missing Ingredient for Fast Stylization
• Self-Normalizing Neural Networks
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
• All you need is a good init
• Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks
• On the Effects of Batch and Weight Normalization in Generative Adversarial Networks
• Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classification
• Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes
• Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning
• Feed-forward Uncertainty Propagation in Belief and Neural Networks

Cited by in corpus (5)

• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• Iterative Normalization: Beyond Standardization towards Efficient Whitening
• Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence
• Path Sample-Analytic Gradient Estimators for Stochastic Binary Networks
• Feed-forward Uncertainty Propagation in Belief and Neural Networks