9 papers · 1 filter
Rethinking AI Cloud Infrastructure for Agentic Serving Systems with the Aries Experimentation Framework
Leonid Kondrashov, Hongrui Liu, JooYoung Park +14
Autonomous agents challenge conventional LLM serving by coupling repeated inference with persistent context and sandboxed tool execution. We present Aries, a full-stack experimenta…
FASER: Fine-Grained Phase Management for Speculative Decoding in Dynamic LLM Serving
Wenyan Chen, Chengzhi Lu, Yanying Lin +1
Speculative decoding (SD) is a widely used approach for accelerating decode-heavy LLM inference workloads. While online inference workloads are highly dynamic, existing SD systems…
Nexus: Transparent I/O Offloading for High-Density Serverless Computing
JooYoung Park, Kevin Nguetchouang, Jovan Stojkovic +4
Serverless computing relies on extreme multi-tenancy to remain economically viable, driving providers to rely on virtual machines (VMs) that ensure strong isolation and seamless ec…
CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inference
Yulin Zou, Yan Chen, Wenyan Chen +5
Video streaming analytics is a crucial workload for vision-language model serving, but the high cost of multimodal inference limits scalability. Prior systems reduce inference cost…
PromptTuner: SLO-Aware Elastic System for LLM Prompt Tuning
Wei Gao, Peng Sun, Dmitrii Ustiugov +2
Prompt tuning has become a prominent strategy for enhancing the performance of Large Language Models (LLMs) on downstream tasks. Many IT enterprises now offer Prompt-Tuning-as-a-Se…
Melding the Serverless Control Plane with the Conventional Cluster Manager for Speed and Resource Efficiency
Leonid Kondrashov, Lazar CvetkoviÄ, Hancheng Wang +2
Serverless platforms face a trade-off: conventional cluster managers like Kubernetes offer compatibility for co-locating Function-as-a-Service (FaaS) and Backend-as-a-Service (BaaS…