On the Regularity of Attention

James Vuckovic, Aristide Baratin, Remi Tachet des Combes

Attention is a powerful component of modern neural networks across a wide variety of domains. In this paper, we seek to quantify the regularity (i.e. the amount of smoothness) of t…