Which Question Is Your Attention Metric Answering? Attention Rows as Compositional Data

Marios Papamichalis, Regina Ruane

Each row of a transformer's attention matrix is a probability distribution over tokens, and in trained models most of that probability lands on a single \emph{sink} token, usually…