Efficient Attention via Pre-Scoring: Prioritizing Informative Keys in Transformers

Zhexiang Li, Haoyu Wang, Yutong Bao +1

Efficient attention mechanisms enable long-context transformers but often miss globally important tokens, degrading modeling quality. We introduce a pre-scoring framework that assi…