A Simple and Effective Positional Encoding for Transformers

Pu-Chin Chen, Henry Tsai, Srinadh Bhojanapalli +3

Transformer models are permutation equivariant. To supply the order and type information of the input tokens, position and segment embeddings are usually added to the input. Recent…