Exploiting Biases in Attention for Transformers

Charles Herr, Gabriele Spadaro, Enzo Tartaglione

Attention is known to be sparse in vision transformers. In particular, we analyze the unequal relevance of attention-matrix columns to the final computation and show that this phen…