SaluNet: Enabling Total Plasticity in Normalization-Free Deep Networks

Mourad Zaied

Normalization layers such as BatchNorm and LayerNorm have long been considered essential for stable training in deep networks. This work demonstrates that they can be fully replace…