A Framework for Non-Linear Attention via Modern Hopfield Networks

Ahmed Farooq

In this work we propose an energy functional along the lines of Modern Hopfield Networks (MNH), the stationary points of which correspond to the attention due to Vaswani et al. [12…