QK-Normed MLA: QK normalization without full key caching

Yizhou Han, Yao Zhao, Jun Zhou +2

Query-key (QK) normalization stabilizes attention by controlling the scale of queries and keys before the dot product, but is not immediately compatible with Multi-head Latent Atte…