SeqPE: Transformer with Sequential Position Encoding

Huayang Li, Yahui Liu, Hongyu Sun +5

Since self-attention layers in Transformers are permutation invariant by design, positional encodings must be explicitly incorporated to enable spatial understanding. However, fixe…