Delta Attention Residuals

Cheng Luo, Zefan Cai, Junjie Hu

Attention Residuals replace standard additive residual connections with learned softmax attention over previous layer outputs, enabling selective cross-layer routing. However, stan…