Interface-Aware KV Cache Quantization for Dense On-Chip NVM in Long-Context LLM Decoding

Jiahao Zheng, Yifan Qin, Xiaobo Sharon Hu +1

The key-value (KV) cache is the dominant memory bottleneck in long-context large language model (LLM) decoding: every step reads it entirely, so decoding is memory-bandwidth bound.…