11 papers
Learning Agent Execution for KV-Cache Management in Agentic Serving
Rui Zhang, Chaeeun Kim, Shaoting Feng +6
Multi-agent LLM systems have emerged as an important deployment paradigm for AI services, where each user request is decomposed into a sequence of specialized agents. Across these…
VeriCache: Turning Lossy KV Cache into Lossless LLM Inference
Jiayi Yao, Samuel Shen, Kuntai Du +7
The large size of the KV cache has become a major bottleneck for serving LLMs with increasing context lengths. In response, many KV cache compression methods, such as token droppin…
The 1/W Law: An Analytical Study of Context-Length Routing Topology and GPU Generation Gains for LLM Inference Energy Efficiency
Huamin Chen, Xunzhuo Liu, Yuhan Liu +3
How many tokens can a GPU inference cluster deliver per watt? Across deployments of identical hardware, the answer varies by 40x -- not because of software inefficiency, but becaus…
The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Project
Huamin Chen, Xunzhuo Liu, Bowei He +5
Over the past year, the vLLM Semantic Router project has released a series of work spanning: (1) core routing mechanisms -- signal-driven routing, context-length pool routing, rout…
FleetOpt: Analytical Fleet Provisioning for LLM Inference with Compress-and-Route as Implementation Mechanism
Huamin Chen, Xunzhuo Liu, Yuhan Liu +3
Modern LLM GPU fleets are provisioned for worst-case context lengths that the vast majority of requests never approach, wasting GPU capacity on idle KV-cache slots. We present Flee…
inference-fleet-sim: A Queueing-Theory-Grounded Fleet Capacity Planner for LLM Inference
Huamin Chen, Xunzhuo Liu, Yuhan Liu +3
Sizing a GPU fleet for LLM inference is harder than it looks. The obvious questions -- how many GPUs, which type, where to split a two-pool fleet -- have no closed-form answers. Th…