most citedKVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization

9 citations · 10 across the 3 of their papers we have counts for

collaborators
Showing cs.LGShow all

1 paper · 1 filter

cs.LG2024★ 9 cited

KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization

Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh +4

LLMs are seeing growing use for applications which require large context windows, and with these large context windows KV cache activations surface as the dominant contributor to m…