Is Batch Norm unique? An empirical investigation and prescription to emulate the best properties of common normalizers without batch dependence
arXiv:2010.10687
Abstract
We perform an extensive empirical study of the statistical properties of Batch Norm and other common normalizers. This includes an examination of the correlation between representations of minibatches, gradient norms, and Hessian spectra both at initialization and over the course of training. Through this analysis, we identify several statistical properties which appear linked to Batch Norm's superior performance. We propose two simple normalizers, PreLayerNorm and RegNorm, which better match these desirable properties without involving operations along the batch dimension. We show that PreLayerNorm and RegNorm achieve much of the performance of Batch Norm without requiring batch dependence, that they reliably outperform LayerNorm, and that they can be applied in situations where Batch Norm is ineffective.
References in corpus (6)
- Layer Normalization
- Axial Attention in Multidimensional Transformers
- Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
- An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
- The large learning rate phase of deep learning: the catapult mechanism
- The Break-Even Point on Optimization Trajectories of Deep Neural Networks