A Manifold Representation of the Key in Vision Transformers

Li Meng, Morten Goodwin, Anis Yazidi +1

Vision Transformers implement multi-head self-attention via stacking multiple attention blocks. The query, key, and value are often intertwined and generated within those blocks vi…