Attention's forward pass and Frank-Wolfe

Albert Alcalde, Borjan Geshkovski, Domènec Ruiz-Balet

We study the hardmax limit of self-attention dynamics for token embeddings obtained in the zero-temperature ($\beta \to +\infty$) regime, and relate it to the finite-$\beta$ setting. In this…