Static Key Attention in Vision

Zizhao Hu, Xiaolin Zhou, Mohammad Rostami

The success of vision transformers is widely attributed to the expressive power of their dynamically parameterized multi-head self-attention mechanism. We examine the impact of sub…