Showing cs.AIShow all
3 papers · 1 filter
cs.AI2026
Where Should the KV Cache Live? Placement Policies Across GPU, CPU, and SSD for Long-Lived Sessions
Srikanta Datta Tumkur, Jay Iyer, Mehar Simhadri +3
GPU high bandwidth memory is scarce and expensive, and KV caches consume much of it as chats, agent loops, and document question answering accumulate state. Systems such as Mooncak…
cs.AI2026
Calibrate, Then Route: A Measured Study of Learned Request Routing for Disaggregated LLM Serving
Srikanta Datta Tumkur, Jay Iyer, Mehar Simhadri +3
Disaggregated LLM serving places compute heavy prefill and memory heavy decode on separate GPU pools. Systems such as DistServe, Splitwise, and Mooncake make this separation fast,…
cs.AI2026
More GPUs or a Smaller Cache? Tensor Parallelism versus KV Compression for Memory-Bound LLM Serving
Srikanta Datta Tumkur, Mehar Simhadri, Anshu Bansal +5
When an LLM serving deployment runs out of KVcache room, there are two well-established ways out. Tensor parallelism shards the weights and the KV cache across two, four, or eight…