ToSA: Token Selective Attention for Efficient Vision Transformers

Manish Kumar Singh, Rajeev Yasarla, Hong Cai +2

In this paper, we propose a novel token selective attention approach, ToSA, which can identify tokens that need to be attended as well as those that can skip a transformer layer. M…