Attention's forward pass and Frank-Wolfe

Albert Alcalde, Borjan Geshkovski, Domènec Ruiz-Balet

We study the hardmax limit of self-attention dynamics for token embeddings obtained in the zero-temperature (${\hat{I}}^2\to +\infty$) regime, and relate it to the finite-${\hat{I}}^2$ setting. In th…