1 citations · 1 across the 3 of their papers we have counts for
7 papers
AM-Bench: Understanding Agents under Task Underspecification across Models and Harnesses
Kaiyu Huang, Yiting Li, Angyang Gu +3
Ambiguous or incomplete task specifications complicate both execution and clarification for language-model agents. Yet agents' responses to these information gaps remain poorly cha…
Semantics, Workflows, and Infrastructure: Understanding Agent Serving at Production Scale
Yihao Zheng, Jingzhe Jiang, Dejiang Zhu +4
Large language model (LLM) agents execute applications through a workflow of inference requests with tool calls and user interactions. Serving these applications at production scal…
PackServe: SLO-Aware Request Scheduling for Agentic LLM Serving at Scale
Zhiyuan Tan, Dejiang Zhu, Jingzhe Jiang +4
Request scheduling is a key challenge in large-scale clusters serving agentic large language model (LLM) workloads. An effective scheduler must preserve key-value cache (KVC) reuse…
PixelFlow: Token-Level Workload Management for Efficient Distributed DiT Serving
Zhexiang Zhang, Minchen Yu, Yifan Sun +3
Online image generation with Diffusion Transformers (DiTs) must meet latency service-level objectives (SLOs) while using GPU resources efficiently. Existing systems improve GPU uti…
UniScale: Adaptive Unified Inference Scaling via Online Joint Optimization of Model Routing and Test-Time Scaling
Kaiyu Huang, Xingyu Wang, Mingze Kong +6
In real-world deployments of large language models (LLMs), balancing inference quality and computational cost has become a central challenge. Existing approaches tackle this trade-…
GoodServe: Towards High-Goodput Serving of Agentic LLM Inferences over Heterogeneous Resources
Boxiao Du, Boning Huangfu, Yizhou Luo +5
Large Language Models (LLMs) play a critical role in emerging agentic applications, where the timely completion of each entire inference is critical. Meanwhile, agentic LLM inferen…
AdaSpec: Adaptive Speculative Decoding for Fast, SLO-Aware Large Language Model Serving
Kaiyu Huang, Hao Wu, Zhubo Shi +3
Cloud-based Large Language Model (LLM) services often face challenges in achieving low inference latency and meeting Service Level Objectives (SLOs) under dynamic request patterns.…