Ablate and Rescue: A Causal Analysis of Residual Stream Hyper-Connections

William Peng, Josheev Rai, Kevin Tseng +2

Multi-stream transformer architectures have recently been proposed as a promising direction for managing representation collapse and the vanishing gradient problem for residual con…