Localmax dynamics for attention in transformers and its asymptotic behavior

Henri Cimetière, Maria Teresa Chiri, Bahman Gharesifard

We introduce a new discrete-time attention model, termed the localmax dynamics, which interpolates between the classic softmax dynamics and the hardmax dynamics, where only the tok…