Dual Dimensionality for Local and Global Attention

Zhiyuan Wang, Xuan Luo, Sirui Zeng +1

Decoder-only Transformers compute attention over the KV cache of preceding tokens. Keys (and Values) are typically represented with the same dimensionality, regardless of its dista…