TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inference

Gradwell Dzikanyanga, Weihao Yang, Hao Huang +4

Key-value (KV) caching is critical for efficient inference in large language models (LLMs), yet its memory footprint scales linearly with context length, resulting in a severe scal…