Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagation

Bobby He, James Martens, Guodong Zhang +4

Skip connections and normalisation layers form two standard architectural components that are ubiquitous for the training of Deep Neural Networks (DNNs), but whose precise roles ar…