Attention Sinks and Outliers in Attention Residuals

Haozheng Luo, Haoran Dai, Shaoyang Zhang +10

We propose OASIS, an outlier- and sink-aware method that stabilizes dual-normalized attention-residual architectures through explicit null routing and token-to-depth null coupling.…