Decoupling the "What" and "Where" With Polar Coordinate Positional Embeddings

Anand Gopalakrishnan, Robert Csordás, Jürgen Schmidhuber +1

The attention mechanism in a Transformer architecture matches key to query based on both content -- the what -- and position in a sequence -- the where. We present an analysis indi…