QCQA: Quality and Capacity-aware grouped Query Attention

Vinay Joshi, Prashant Laddha, Shambhavi Sinha +2

Excessive memory requirements of key and value features (KV-cache) present significant challenges in the autoregressive inference of large language models (LLMs), restricting both…