Intrinsically Interpretable Attention via Sparse Post-Training

Florent Draye, Anson Lei, Hsiao-Ru Pan +2

We introduce a simple post-training method that makes transformer attention sparse without sacrificing performance. Applying a flexible sparsity regularisation under a constrained-…