Do Vision Transformers Need All-to-All Attention? Global Communication Through Elastic Learned Cores

Alan Z. Song, Yinjie Chen, Mu Nan +8

Vision Transformers (ViTs) learn rich visual-semantic representations through all-to-all self-attention among patch tokens. However, this design implicitly assumes that direct pair…