11 papers
Rethinking AI Cloud Infrastructure for Agentic Serving Systems with the Aries Experimentation Framework
Leonid Kondrashov, Hongrui Liu, JooYoung Park +14
Autonomous agents challenge conventional LLM serving by coupling repeated inference with persistent context and sandboxed tool execution. We present Aries, a full-stack experimenta…
FASER: Fine-Grained Phase Management for Speculative Decoding in Dynamic LLM Serving
Wenyan Chen, Chengzhi Lu, Yanying Lin +1
Speculative decoding (SD) is a widely used approach for accelerating decode-heavy LLM inference workloads. While online inference workloads are highly dynamic, existing SD systems…
Nexus: Transparent I/O Offloading for High-Density Serverless Computing
JooYoung Park, Kevin Nguetchouang, Jovan Stojkovic +4
Serverless computing relies on extreme multi-tenancy to remain economically viable, driving providers to rely on virtual machines (VMs) that ensure strong isolation and seamless ec…
CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inference
Yulin Zou, Yan Chen, Wenyan Chen +5
Video streaming analytics is a crucial workload for vision-language model serving, but the high cost of multimodal inference limits scalability. Prior systems reduce inference cost…
PromptTuner: SLO-Aware Elastic System for LLM Prompt Tuning
Wei Gao, Peng Sun, Dmitrii Ustiugov +2
Prompt tuning has become a prominent strategy for enhancing the performance of Large Language Models (LLMs) on downstream tasks. Many IT enterprises now offer Prompt-Tuning-as-a-Se…
MemTrust: A Zero-Trust Architecture for Unified AI Memory System
Xing Zhou, Dmitrii Ustiugov, Haoxin Shang +1
AI memory systems are evolving toward unified context layers that enable efficient cross-agent collaboration and multi-tool workflows, facilitating better accumulation of personal…