Learning interpretable positional encodings in transformers depends on initialization

Takuya Ito, Luca Cocchi, Tim Klinger +3

In transformers, the positional encoding (PE) provides essential information that distinguishes the position and order amongst tokens in a sequence. Most prior investigations of PE…