Attention Graphons: A Graph Limit Perspective on Graph Transformers

Caio F. Deberaldini Netto, Moshe Eliasof, Luana Ruiz

Graph Transformers produce, for each attention head, a dense $n\times n$ matrix of learned pairwise interactions. We ask a fundamental question: do these attention-induced graphs c…