On the Mathematical Relationship Between Layer Normalization and Dynamic Activation Functions

Felix Stollenwerk

Layer normalization (LN) is an essential component of modern neural networks. While many alternative techniques have been proposed, none of them have succeeded in replacing LN so f…