Fast Monte-Carlo Approximation of the Attention Mechanism

Hyunjun Kim, JeongGil Ko

We introduce Monte-Carlo Attention (MCA), a randomized approximation method for reducing the computational cost of self-attention mechanisms in Transformer architectures. MCA explo…