Attention Condensation via Sparsity Induced Regularized Training

Eli Sason, Darya Frolova, Boris Nazarov +1

As the context window expands, self-attention increasingly dominates the transformer's inference time. Therefore, accelerating attention computation while minimizing performance de…