Geometric Interpretation of Layer Normalization and a Comparative Analysis with RMSNorm

Akshat Gupta, Atahan Ozdemir, Gopala Anumanchipalli

This paper presents a novel geometric interpretation of LayerNorm and explores how LayerNorm influences the norm and orientation of hidden vectors in the representation space. With…