SeeDNorm: Self-Rescaled Dynamic Normalization

Wenrui Cai, Defa Zhu, Qingjie Liu +1

Normalization layer constitutes an essential component in neural networks. In transformers, the predominantly used RMSNorm constrains vectors to a unit hypersphere, followed by dim…