Understanding and Improving Layer Normalization

Abstract

Layer normalization (LayerNorm) is a technique to normalize the distributions of intermediate layers. It enables smoother gradients, faster training, and better generalization accuracy. However, it is still unclear where the effectiveness stems from. In this paper, our main contribution is to take a step further in understanding LayerNorm. Many of previous studies believe that the success of LayerNorm comes from forward normalization. Unlike them, we find that the derivatives of the mean and variance are more important than forward normalization by re-centering and re-scaling backward gradients. Furthermore, we find that the parameters of LayerNorm, including the bias and gain, increase the risk of over-fitting and do not work in most cases. Experiments show that a simple version of LayerNorm (LayerNorm-simple) without the bias and gain outperforms LayerNorm on four datasets. It obtains the state-of-the-art performance on En-Vi machine translation. To address the over-fitting problem, we propose a new normalization method, Adaptive Normalization (AdaNorm), by replacing the bias and gain with a new transformation function. Experiments show that AdaNorm demonstrates better results than LayerNorm on seven out of eight datasets.

Accepted by NeurIPS 2019

Cited by in corpus (24)

• A Survey on Visual Transformer
• Review: Deep Learning in Electron Microscopy
• Very Deep Transformers for Neural Machine Translation
• Incorporating Dynamic Semantics into Pre-Trained Language Model for Aspect-based Sentiment Analysis
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• SODFormer: Streaming Object Detection with Transformer Using Events and Frames
• Transformer-based Map Matching Model with Limited Ground-Truth Data using Transfer-Learning Approach
• AdaTerm: Adaptive T-Distribution Estimated Robust Moments for Noise-Robust Stochastic Gradient Optimization
• Supervised Contrastive Learning based Dual-Mixer Model for Remaining Useful Life Prediction
• Dropout Q-Functions for Doubly Efficient Reinforcement Learning
• PowerNorm: Rethinking Batch Normalization in Transformers
• Rethinking Skip Connection with Layer Normalization in Transformers and ResNets
• Batch Group Normalization
• Adversarially Adaptive Normalization for Single Domain Generalization
• Batch Normalization Preconditioning for Neural Network Training
• Robust Optimization for Multilingual Translation with Imbalanced Data
• Predicting small molecules solubilities on endpoint devices using deep ensemble neural networks
• Implicit Regularization and Convergence for Weight Normalization
• Correct Normalization Matters: Understanding the Effect of Normalization On Deep Neural Network Models For Click-Through Rate Prediction
• Query-Key Normalization for Transformers
• When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute
• Group Whitening: Balancing Learning Efficiency and Representational Capacity
• A Survey on Green Deep Learning
• STIL -- Simultaneous Slot Filling, Translation, Intent Classification, and Language Identification: Initial Results using mBART on MultiATIS++