Virtual Head Attention

Xibo Ding, Guoxia Wang, Jinle Zeng +4

The design of attention mechanisms in long-context large language models faces an ``impossible triangle'' among KV cache size, training and prefill FLOPs, and model quality. GQA ac…