Rethinking Caching for LLM Serving Systems: Beyond Traditional Heuristics

Jungwoo Kim, Minsang Kim, Jaeheon Lee +6

Serving Large Language Models (LLMs) at scale requires meeting strict Service Level Objectives (SLOs) under severe computational and memory constraints. Nevertheless, traditional c…