AutoNorm: Understanding Adaptive Normalization in Transformers through Differentiable Gating

Piyush Kaushik Bhattacharyya, Divyanshu Rai, Swastik Singh +3

Normalization is a critical component for stabilizing Transformer training, yet the choice between static strategies such as Layer Normalization (LN) and adaptive alternatives rema…