Switching Linear Attention

Hyun Dong Lee, Xavier Gonzalez, Nicolas Zucchet +3

Designing expressive sequence layers with efficient inference remains a central challenge in modern machine learning. Standard softmax attention achieves excellent sequence modelin…