Representative Attention For Vision Transformers

Yuntong Li, Hainuo Wang, Hengxing Liu +2

Linear attention has emerged as a promising direction for scaling Vision Transformers beyond the quadratic cost of dense self-attention. A prevalent strategy is to compress spatial…