General Non-Clairvoyant KV-Cache Scheduling via Regime-Aware Routing

Yiding Feng, Siyu Liu, Zonghan Yang +1

We study non-clairvoyant scheduling for batched Large Language Model (LLM) inference under a hard Key-Value (KV) cache memory budget. Each request has a known prompt length but an…