4 papers · 1 filter
The Inference Engineering Pareto Atlas: Which Optimizations Dominate the Cost, Quality, and Latency Frontier?
Srikanta Datta Tumkur, Jay Iyer, Mehar Simhadri +3
LLM inference optimizations report speedups on different models, GPUs, prompts, and quality metrics, making them hard to compare or combine. We build a cost, quality, and latency P…
Where Should the KV Cache Live? Placement Policies Across GPU, CPU, and SSD for Long-Lived Sessions
Srikanta Datta Tumkur, Jay Iyer, Mehar Simhadri +3
GPU high bandwidth memory is scarce and expensive, and KV caches consume much of it as chats, agent loops, and document question answering accumulate state. Systems such as Mooncak…
Calibrate, Then Route: A Measured Study of Learned Request Routing for Disaggregated LLM Serving
Srikanta Datta Tumkur, Jay Iyer, Mehar Simhadri +3
Disaggregated LLM serving places compute heavy prefill and memory heavy decode on separate GPU pools. Systems such as DistServe, Splitwise, and Mooncake make this separation fast,…
More GPUs or a Smaller Cache? Tensor Parallelism versus KV Compression for Memory-Bound LLM Serving
Srikanta Datta Tumkur, Mehar Simhadri, Anshu Bansal +5
When an LLM serving deployment runs out of KVcache room, there are two well-established ways out. Tensor parallelism shards the weights and the KV cache across two, four, or eight…