Top-Theta Attention: Sparsifying Transformers by Compensated Thresholding

Konstantin Berestizshevsky, Renzo Andri, Lukas Cavigelli

We present Top-Theta (Top-$\theta$) Attention, a training-free method for sparsifying transformer attention during inference. Our key insight is that static, per-head thresholds can be…