Cottention: Linear Transformers With Cosine Attention

Gabriel Mongaras, Trevor Dohm, Eric C. Larson

Attention mechanisms, particularly softmax attention, have been instrumental in the success of transformer-based models such as GPT. However, the quadratic memory complexity of sof…