Oneiros: KV Cache Optimization through Parameter Remapping for Multi-tenant LLM Serving

Ruihao Li, Shagnik Pal, Vineeth Narayan Pullu +4

KV cache accelerates LLM inference by avoiding redundant computation, at the expense of memory. To support larger KV caches, prior work extends GPU memory with CPU memory via CPU-o…