Enjoy Your Layer Normalization with the Computational Efficiency of RMSNorm

Yuxin Guo, Yihao Yue, Yunhao Ni +4

Layer normalization (LN) is a fundamental component in modern deep learning, but its per-sample centering and scaling introduce non-negligible inference overhead. RMSNorm improves…