Is Batch Norm unique? An empirical investigation and prescription to emulate the best properties of common normalizers without batch dependence
arXiv:2010.10687
Abstract
We perform an extensive empirical study of the statistical properties of Batch Norm and other common normalizers. This includes an examination of the correlation between representations of minibatches, gradient norms, and Hessian spectra both at initialization and over the course of training. Through this analysis, we identify several statistical properties which appear linked to Batch Norm's superior performance. We propose two simple normalizers, PreLayerNorm and RegNorm, which better match these desirable properties without involving operations along the batch dimension. We show that PreLayerNorm and RegNorm achieve much of the performance of Batch Norm without requiring batch dependence, that they reliably outperform LayerNorm, and that they can be applied in situations where Batch Norm is ineffective.
References in corpus (11)
- Instance Normalization: The Missing Ingredient for Fast Stylization
- Layer Normalization
- Optimal whitening and decorrelation
- Axial Attention in Multidimensional Transformers
- Exponential expressivity in deep neural networks through transient chaos
- Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
- An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
- Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks
- The large learning rate phase of deep learning: the catapult mechanism
- The Break-Even Point on Optimization Trajectories of Deep Neural Networks
- The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent