Effective Theory of Transformers at Initialization

Emily Dinan, Sho Yaida, Susan Zhang

We perform an effective-theory analysis of forward-backward signal propagation in wide and deep Transformers, i.e., residual neural networks with multi-head self-attention blocks a…