Unveiling and Controlling Anomalous Attention Distribution in Transformers

Ruiqing Yan, Xingbo Du, Haoyu Deng +7

With the advent of large models based on the Transformer architecture, researchers have observed an anomalous phenomenon in the Attention mechanism--there is a very high attention…