Attention Sinks and Outliers in Attention Residuals

Haozheng Luo, Haoran Dai, Shaoyang Zhang +10

We propose OASIS, an outlier- and sink-aware technique built on inter-layer null signaling. As AttnResidual architectures introduce an additional depth-wise normalization channel,…