161 citations · 162 across the 11 of their papers we have counts for
9 papers · 1 filter
Rethinking AI Cloud Infrastructure for Agentic Serving Systems with the Aries Experimentation Framework
Leonid Kondrashov, Hongrui Liu, JooYoung Park +14
Autonomous agents challenge conventional LLM serving by coupling repeated inference with persistent context and sandboxed tool execution. We present Aries, a full-stack experimenta…
FASER: Fine-Grained Phase Management for Speculative Decoding in Dynamic LLM Serving
Wenyan Chen, Chengzhi Lu, Yanying Lin +1
Speculative decoding (SD) is a widely used approach for accelerating decode-heavy LLM inference workloads. While online inference workloads are highly dynamic, existing SD systems…
Nexus: Transparent I/O Offloading for High-Density Serverless Computing
JooYoung Park, Kevin Nguetchouang, Jovan Stojkovic +4
Serverless computing relies on extreme multi-tenancy to remain economically viable, driving providers to rely on virtual machines (VMs) that ensure strong isolation and seamless ec…
CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inference
Yulin Zou, Yan Chen, Wenyan Chen +5
Video streaming analytics is a crucial workload for vision-language model serving, but the high cost of multimodal inference limits scalability. Prior systems reduce inference cost…
PromptTuner: SLO-Aware Elastic System for LLM Prompt Tuning
Wei Gao, Peng Sun, Dmitrii Ustiugov +2
Prompt tuning has become a prominent strategy for enhancing the performance of Large Language Models (LLMs) on downstream tasks. Many IT enterprises now offer Prompt-Tuning-as-a-Se…
TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocity
Ruiqi Lai, Hongrui Liu, Chengzhi Lu +6
The architectural shift to prefill/decode (PD) disaggregation in LLM serving improves resource utilization but struggles with the bursty nature of modern workloads. Existing autosc…