Subcritical Signal Propagation at Initialization in Normalization-Free Transformers

Sergey Alekseev

We study signal propagation at initialization in transformers through the averaged partial Jacobian norm (APJN), a measure of gradient amplification across layers. We extend APJN a…