Gated Normalization Removal and Scale Anchoring in Pre-Norm Transformers

Andrei Kanavalau, Carmen Amo Alonso, Sanjay Lall

Normalization layers are standard in transformers, but it is not clear whether their sample-dependent computations are necessary throughout both training and inference. This work d…