Fast attention mechanisms: a tale of parallelism

Jingwen Liu, Hantao Yu, Clayton Sanford +2

Transformers have the representational capacity to simulate Massively Parallel Computation (MPC) algorithms, but they suffer from quadratic time complexity, which severely limits t…