Root Mean Square Layer Normalization

Abstract

Layer normalization (LayerNorm) has been successfully applied to various deep neural networks to help stabilize training and boost model convergence because of its capability in handling re-centering and re-scaling of both inputs and weight matrix. However, the computational overhead introduced by LayerNorm makes these improvements expensive and significantly slows the underlying network, e.g. RNN in particular. In this paper, we hypothesize that re-centering invariance in LayerNorm is dispensable and propose root mean square layer normalization, or RMSNorm. RMSNorm regularizes the summed inputs to a neuron in one layer according to root mean square (RMS), giving the model re-scaling invariance property and implicit learning rate adaptation ability. RMSNorm is computationally simpler and thus more efficient than LayerNorm. We also present partial RMSNorm, or pRMSNorm where the RMS is estimated from p% of the summed inputs without breaking the above properties. Extensive experiments on several tasks using diverse network architectures show that RMSNorm achieves comparable performance against LayerNorm but reduces the running time by 7%~64% on different models. Source code is available at https://github.com/bzhangGo/rmsnorm.

NeurIPS 2019

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Teaching Machines to Read and Comprehend
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Order-Embeddings of Images and Language
• Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks
• Recurrent Batch Normalization
• Syllable-Based Sequence-to-Sequence Speech Recognition with the Transformer in Mandarin Chinese

Cited by in corpus (8)

• Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmap
• A Comprehensive Review of State-of-The-Art Methods for Java Code Generation from Natural Language Text
• Transfer Learning with Foundational Models for Time Series Forecasting using Low-Rank Adaptations
• Tucano: Advancing Neural Text Generation for Portuguese
• Improving Neural Network Training using Dynamic Learning Rate Schedule for PINNs and Image Classification
• Towards Foundation Models for Materials Science: The Open MatSci ML Toolkit
• Large Language Models -- the Future of Fundamental Physics?
• LiRA: Light-Robust Adversary for Model-based Reinforcement Learning in Real World