From Sparsity to Simplicity: Enabling Simpler Sequential Replacements via Sparse Attention Distillation

Yuxin Ren, Maxwell D Collins, Miao Hu +1

Self-attention serves as the core foundation of large-scale transformer pretraining, but its quadratic token interaction cost makes inference expensive. Replacing attention with si…