On the Invariants of Softmax Attention

Wonsuk Lee

Softmax attention maps every query--key interaction into a probability distribution, but the underlying structure remains largely unexplored. We define the \emph{energy field}, the…