Recency/Frequency Adaptive KV Caching for Large Language Model Serving

Yang Shen, Meghana Madhyastha, Robert Underwood +2

Key-value (KV) caching is a powerful technique for accelerating large language model inference and generation. Inference workloads are large and diverse, which makes them difficult…