Geometry without Position? When Positional Embeddings Help and Hurt Spatial Reasoning

Jian Shi, Michael Birsak, Wenqing Cui +2

This paper revisits the role of positional embeddings (PEs) within vision transformers (ViTs) from a geometric perspective. We show that PEs are not mere token indices but effectiv…