10 citations · 23 across the 16 of their papers we have counts for
12 papers · 1 filter
SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling
Jiahao Wang, Kaizhan Lin, Kaixi Zhang +7
LLM scheduling is critical to serving, yet how well existing designs fit agentic serving--where agents, not humans, issue the requests--remains unclear. Agents shift the workload i…
CoAgent: Concurrency Control for Multi-Agent Systems
Hongtao Lyu, Dingyan Zhang, Mingyu Wu +2
Multi-agent LLM systems -- coding agents, devops agents, document agents -- now routinely run several agents in parallel against the same git tree, Kubernetes cluster, or document.…
Simple is Better: Multiplication May Be All You Need for LLM Request Scheduling
Dingyan Zhang, Jinbo Han, Kaixi Zhang +6
High-quality LLM request scheduling requires meeting two key objectives: ensuring the routed instance has KVCache to accelerate request execution, and ensuring that the workload is…
Fast LLM Post-training via Decoupled and Fastest-of-N Speculation
Rongxin Cheng, Kai Zhou, Xingda Wei +8
Rollout dominates the training time in large language model (LLM) post-training, where the trained model is used to generate tokens given a batch of prompts. This work, SpecActor,…
KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider
Jiahao Wang, Jinbo Han, Xingda Wei +6
Serving large language models (LLMs) is important for cloud providers, and caching intermediate results (KV$) after processing each request substantially improves serving throughp…
DiFache: Efficient and Scalable Caching on Disaggregated Memory using Decentralized Coherence
Hanze Zhang, Kaiming Wang, Rong Chen +2
The disaggregated memory (DM) architecture offers high resource elasticity at the cost of data access performance. While caching frequently accessed data in compute nodes (CNs) red…
DecLock: A Case of Decoupled Locking for Disaggregated Memory
Hanze Zhang, Ke Cheng, Rong Chen +2
This paper reveals that locking can significantly degrade the performance of applications on disaggregated memory (DM), sometimes by several orders of magnitude, due to contention…
KunServe: Parameter-centric Memory Management for Efficient Memory Overloading Handling in LLM Serving
Rongxin Cheng, Yuxin Lai, Xingda Wei +2
Serving LLMs with a cluster of GPUs is common nowadays, where the serving system must meet strict latency SLOs required by applications. However, the stateful nature of LLM serving…
BLITZSCALE: Fast and Live Large Model Autoscaling with O(1) Host Caching
Dingyan Zhang, Haotian Wang, Yang Liu +4
Model autoscaling is the key mechanism to achieve serverless model-as-a-service, but it faces a fundamental trade-off between scaling speed and storage/memory usage to cache parame…
Characterizing the Dilemma of Performance and Index Size in Billion-Scale Vector Search and Breaking It with Second-Tier Memory
Rongxin Cheng, Yifan Peng, Xingda Wei +4
Vector searches on large-scale datasets are critical to modern online services like web search and RAG, which necessity storing the datasets and their index on the secondary storag…
PhoenixOS: Concurrent OS-level GPU Checkpoint and Restore with Validated Speculation
Xingda Wei, Zhuobin Huang, Tianle Sun +5
PHOENIXOS (PHOS) is the first OS service that can concurrently checkpoint and restore (C/R) GPU processes--a fundamental capability for critical tasks such as fault tolerance, proc…
Characterizing Off-path SmartNIC for Accelerating Distributed Systems
Xingda Wei, Rongxin Cheng, Yuhan Yang +2
SmartNICs have recently emerged as an appealing device for accelerating distributed systems. However, there has not been a comprehensive characterization of SmartNICs, and existing…