5 papers
A Self-Defining Operator for Distributed Applications
Shihang Li, Thomas Anderson, Ratul Mahajan +2
To operate distributed applications with high availability, operators must rapidly diagnose and mitigate any incident. LLM-based agents can help, but unlike experienced human opera…
CacheWise: Understanding Workloads and Optimizing KVCache Management for Efficiently Serving LLM Coding Agents
Shubham Tiwari, Tapan Chugh, Nash Rickert +3
Coding agents are a fast-growing LLM application, executing as long-running closed-loop sessions in which LLM generations alternate with external tool calls. Yet, unlike chat workl…
Reducing the GPU Memory Bottleneck with Lossless Compression for ML -- Extended
Aditya K Kamath, Arvind Krishnamurthy, Marco Canini +1
Machine learning (ML) training and inference often process data sets far exceeding GPU memory capacity, forcing them to rely on PCIe for on-demand tensor transfers, causing critica…
VibeServe: Can AI Agents Build Bespoke LLM Serving Systems?
Keisuke Kamahori, Shihang Li, Simon Peter +1
For years, we have built LLM serving systems like any other critical infrastructure, with a single general-purpose stack hand-tuned over many engineer-years to support every model…
Cloud abstractions for AI workloads
Marco Canini, Theophilus A. Benson, Ricardo Bianchini +4
AI workloads, often hosted in multi-tenant cloud environments, require vast computational resources but suffer inefficiencies due to limited tenant-provider coordination. Tenants l…