7 papers
Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving
Muhammad Adnan, Rohan Mahapatra, Prashant J. Nair +4
The reasoning and agentic capabilities of large language models have expanded the range of applications they support, from short interactive exchanges to long, compute-heavy reques…
Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scale
Banruo Liu, Haoran Qiu, Ãñigo Goiri +3
AI coding agents like GitHub Copilot, Claude Code, and Codex interleave multi-step LLM inference with tool execution, creating a workload different from chatbots. We present the fi…
Beyond Prediction: Tail-Aware Scheduling for LLM Inference
Yueying Li, Yuanfan Chen, Jiayang Chen +6
LLM serving exhibits extreme length variability, making size-based scheduling difficult in practice. Recent LLM schedulers approximate SJF/SRPT using predicted decode lengths or ra…
Nanvix: A Multikernel OS Design for High-Density Serverless Deployments
Carlos Segarra, Pedro Henrique Penna, Enrique Saurez +4
Serverless providers strive for high resource utilization by optimizing deployment density: how many applications can be deployed per host server. However, achieving high deploymen…
StreamWise: Serving Multi-Modal Generation in Real-Time at Scale
Haoran Qiu, Gohar Irfan Chaudhry, Chaojie Zhang +4
Advances in multi-modal generative models are enabling new applications, from storytelling to automated media synthesis. Most current workloads generate simple outputs (e.g., image…
ModServe: Modality- and Stage-Aware Resource Disaggregation for Scalable Multimodal Model Serving
Haoran Qiu, Anish Biswas, Zihan Zhao +9
Large multimodal models (LMMs) demonstrate impressive capabilities in understanding images, videos, and audio beyond text. However, efficiently serving LMMs in production environme…