HLA: Hadamard Linear Attention

Hanno Ackermann, Hong Cai, Mohsen Ghafoorian +1

The attention mechanism is an important reason for the success of transformers. It relies on computing pairwise relations between tokens. To reduce the high computational cost of s…