Algebraic Positional Encodings

Konstantinos Kogkalidis, Jean-Philippe Bernardy, Vikas Garg

We introduce a novel positional encoding strategy for Transformer-style models, addressing the shortcomings of existing, often ad hoc, approaches. Our framework provides a flexible…