Parabolic Position Encoding: Vision-Centric, Principled, Extrapolatable, General

Christoffer Koo Øhrstrøm, Rafael I. Cabral Muchacho, Yifei Dong +4

We propose Parabolic Position Encoding (PaPE), a parabola-based position encoding for vision modalities in attention-based architectures. Given a set of vision tokens-such as from…