Integral Transformer: Denoising Attention, Not Too Much Not Too Little

Ivan Kobyzev, Abbas Ghaddar, Dingtao Hu +1

Softmax self-attention often assigns disproportionate weight to semantically uninformative tokens such as special tokens and punctuation, a phenomenon known as attention noise. Whi…