PowerNorm: Rethinking Batch Normalization in Transformers

Abstract

The standard normalization method for neural network (NN) models used in Natural Language Processing (NLP) is layer normalization (LN). This is different than batch normalization (BN), which is widely-adopted in Computer Vision. The preferred use of LN in NLP is principally due to the empirical observation that a (naive/vanilla) use of BN leads to significant performance degradation for NLP tasks; however, a thorough understanding of the underlying reasons for this is not always evident. In this paper, we perform a systematic study of NLP transformer models to understand why BN has a poor performance, as compared to LN. We find that the statistics of NLP data across the batch dimension exhibit large fluctuations throughout training. This results in instability, if BN is naively implemented. To address this, we propose Power Normalization (PN), a novel normalization scheme that resolves this issue by (i) relaxing zero-mean normalization in BN, (ii) incorporating a running quadratic mean instead of per batch statistics to stabilize fluctuations, and (iii) using an approximate backpropagation for incorporating the running statistics in the forward pass. We show theoretically, under mild assumptions, that PN leads to a smaller Lipschitz constant for the loss, compared with BN. Furthermore, we prove that the approximate backpropagation scheme leads to bounded gradients. We extensively test PN for transformers on a range of NLP tasks, and we show that it significantly outperforms both LN and BN. In particular, PN outperforms LN by 0.4/0.6 BLEU on IWSLT14/WMT14 and 5.6/3.0 PPL on PTB/WikiText-103. We make our code publicly available at \url{https://github.com/sIncerass/powernorm}.

References in corpus (17)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Spectral Normalization for Generative Adversarial Networks
• Instance Normalization: The Missing Ingredient for Fast Stylization
• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• Wide Residual Networks
• Pay Less Attention with Lightweight and Dynamic Convolutions
• Reducing Transformer Depth on Demand with Structured Dropout
• Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
• Understanding and Improving Layer Normalization
• Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
• Fixup Initialization: Residual Learning Without Normalization
• A Tensorized Transformer for Language Modeling
• Learning Deep Transformer Models for Machine Translation
• Representation Degeneration Problem in Training Natural Language Generation Models
• Towards Stabilizing Batch Statistics in Backward Propagation of Batch Normalization
• PyHessian: Neural Networks Through the Lens of the Hessian
• Positional Normalization

Cited by in corpus (3)

• Long-Range Transformers for Dynamic Spatiotemporal Forecasting
• Rethinking Skip Connection with Layer Normalization in Transformers and ResNets
• Correct Normalization Matters: Understanding the Effect of Normalization On Deep Neural Network Models For Click-Through Rate Prediction