How Vision Transformers Represent Position: A Mechanistic Comparison of Absolute and Rotary Positional Encodings

Mahmoud Mannes, Nikhil Maturi, Parva Mehta +2

Positional embeddings encode the spatial position of each image patch in a Vision Transformer (ViT), allowing the model to distinguish different spatial arrangements of visual cont…