Learning to Focus: Focal Attention for Selective and Scalable Transformers

Dhananjay Ram, Wei Xia, Stefano Soatto

Attention is a core component of transformer architecture, whether encoder-only, decoder-only, or encoder-decoder model. However, the standard softmax attention often produces nois…