11 papers
Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooning
Shan Yu, Yifan Qiao, Mingyuan Ma +18
Inference providers must maintain availability for many LLMs, including low-volume but essential models, making resource efficiency increasingly important as token prices fall. Ana…
BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batching
Yilong Zhao, Shuo Yang, Kan Zhu +5
Offline batch inference, which leverages the flexibility of request batching to achieve higher throughput and lower costs, is becoming more popular for latency-insensitive applicat…
R2-Router: A New Paradigm for LLM Routing with Reasoning
Jiaqi Xue, Qian Lou, Jiarong Xing +1
As LLMs proliferate with diverse capabilities and costs, LLM routing has emerged by learning to predict each LLM's quality and cost for a given query, then selecting the one with h…
The Routing Plateau: Understanding and Breaking the Accuracy Limits of LLM Routers
Yifan Lu, Qiyue Zhang, Shenrun Zhang +4
LLM routing has become a popular approach to improve the cost-quality trade-off of LLM services by dynamically selecting a model for each query. Recent work has explored a broad ra…
Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Serving
Shan Yu, Junyi Shu, Yuanjiang Ni +14
As LLM applications grow more complex, developers are increasingly adopting multi-agent architectures to decompose workflows into specialized, collaborative components, introducing…
Let the Barbarians In: How AI Can Accelerate Systems Performance Research
Audrey Cheng, Shu Liu, Melissa Pan +18
Artificial Intelligence (AI) is beginning to transform the research process by automating the discovery of new solutions. This shift depends on the availability of reliable verifie…