2 citations · 7 across the 15 of their papers we have counts for
16 papers
SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling
Jiahao Wang, Kaizhan Lin, Kaixi Zhang +7
LLM scheduling is critical to serving, yet how well existing designs fit agentic serving--where agents, not humans, issue the requests--remains unclear. Agents shift the workload i…
Efficient K-generalizable Learned Search
Yifan Peng, Jiafei Fan, Xingda Wei +7
Learned top-K search improves the accuracy-latency trade-off of graph-based vector search, but existing methods target a fixed K. Serving real-world workloads with various K values…
Towards Fully-fledged GPU Multitasking via Proactive Memory Scheduling
Weihang Shen, Yinqiu Chen, Rong Chen +1
The limited HBM capacity has become the primary bottleneck for hosting an increasing number of larger-scale GPU tasks. While demand paging extends capacity via host DRAM, it incurs…
Fast LLM Post-training via Decoupled and Fastest-of-N Speculation
Rongxin Cheng, Kai Zhou, Xingda Wei +8
Rollout dominates the training time in large language model (LLM) post-training, where the trained model is used to generate tokens given a batch of prompts. This work, SpecActor,…
Holistic Heterogeneous Scheduling for Autonomous Applications using Fine-grained, Multi-XPU Abstraction
Mingcong Han, Weihang Shen, Rong Chen +2
Modern autonomous applications are increasingly utilizing multiple heterogeneous processors (XPUs) to accelerate different stages of algorithm modules. However, existing runtime sy…
Towards Lock Modularization for Heterogeneous Environments
Hanze Zhang, Rong Chen, Haibo Chen
Modern hardware environments are becoming increasingly heterogeneous, leading to the emergence of applications specifically designed to exploit this heterogeneity. Efficiently adop…
KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider
Jiahao Wang, Jinbo Han, Xingda Wei +6
Serving large language models (LLMs) is important for cloud providers, and caching intermediate results (KV$) after processing each request substantially improves serving throughp…
DiFache: Efficient and Scalable Caching on Disaggregated Memory using Decentralized Coherence
Hanze Zhang, Kaiming Wang, Rong Chen +2
The disaggregated memory (DM) architecture offers high resource elasticity at the cost of data access performance. While caching frequently accessed data in compute nodes (CNs) red…
DecLock: A Case of Decoupled Locking for Disaggregated Memory
Hanze Zhang, Ke Cheng, Rong Chen +2
This paper reveals that locking can significantly degrade the performance of applications on disaggregated memory (DM), sometimes by several orders of magnitude, due to contention…
KunServe: Parameter-centric Memory Management for Efficient Memory Overloading Handling in LLM Serving
Rongxin Cheng, Yuxin Lai, Xingda Wei +2
Serving LLMs with a cluster of GPUs is common nowadays, where the serving system must meet strict latency SLOs required by applications. However, the stateful nature of LLM serving…
BLITZSCALE: Fast and Live Large Model Autoscaling with O(1) Host Caching
Dingyan Zhang, Haotian Wang, Yang Liu +4
Model autoscaling is the key mechanism to achieve serverless model-as-a-service, but it faces a fundamental trade-off between scaling speed and storage/memory usage to cache parame…
Microsecond-scale Dynamic Validation of Idempotency for GPU Kernels
Mingcong Han, Weihang Shen, Guanwen Peng +2
We discovered that a GPU kernel can have both idempotent and non-idempotent instances depending on the input. These kernels, called conditionally-idempotent, are prevalent in real-…
Characterizing the Dilemma of Performance and Index Size in Billion-Scale Vector Search and Breaking It with Second-Tier Memory
Rongxin Cheng, Yifan Peng, Xingda Wei +4
Vector searches on large-scale datasets are critical to modern online services like web search and RAG, which necessity storing the datasets and their index on the secondary storag…
PhoenixOS: Concurrent OS-level GPU Checkpoint and Restore with Validated Speculation
Xingda Wei, Zhuobin Huang, Tianle Sun +5
PHOENIXOS (PHOS) is the first OS service that can concurrently checkpoint and restore (C/R) GPU processes--a fundamental capability for critical tasks such as fault tolerance, proc…
Characterizing Network Requirements for GPU API Remoting in AI Applications
Tianxia Wang, Zhuofu Chen, Xingda Wei +3
GPU remoting is a promising technique for supporting AI applications. Networking plays a key role in enabling remoting. However, for efficient remoting, the network requirements in…
KRCORE: a microsecond-scale RDMA control plane for elastic computing
Xingda Wei, Fangming Lu, Rong Chen +1
We present KRCORE, an RDMA library with a microsecond-scale control plane on commodity RDMA hardware for elastic computing. KRCORE can establish a full-fledged RDMA connection with…