6 papers
KVCompass: A Discovery Plane for Internet-Scale Prefix Cache Sharing in LLM Serving
Shiyi Ling, Fei Fang, Chen Qian
KV cache reuse is one of the most effective ways to reduce LLM inference cost, but today its benefits are largely confined to a single cluster. When a provider serves the same mode…
Tessera: Demand-Driven KV Cache Management for Retrieval-Augmented LLM Serving
Fei Fang, Chung-Hsiang Lo, Yi Liu +2
RAG and retrieval-based agent memory both inject retrieved content into LLM prompts, as document chunks and recalled memory records, respectively. The same content can recur across…
d-HNSW: A High-performance Vector Search Engine on Disaggregated Memory
Fei Fang, Yi Liu, Chen Qian
Efficient vector search is essential for powering large-scale AI applications, such as LLMs. Existing solutions are designed for monolithic architectures where compute and memory a…
Trinity: Disaggregating Vector Search from Prefill-Decode Disaggregation in LLM Serving
Yi Liu, Chen Qian
Prefill and decode (PD) disaggregation separates prompt prefill and token-by-token decode stages into distinct GPU pools and has become the dominant architecture for large-scale LL…
Fantasy: Efficient Large-scale Vector Search on GPU Clusters with GPUDirect Async
Yi Liu, Chen Qian
Vector similarity search has become a critical component in AI-driven applications such as large language models (LLMs). To achieve high recall and low latency, GPUs are utilized t…
PartialLoading: User Scheduling and Bandwidth Allocation for Parameter-sharing Edge Inference
Guanqiao Qu, Qian Chen, Xianhao Chen +2
By provisioning inference offloading services, edge inference drives the rapid growth of AI applications at network edge. However, how to reduce the inference latency remains a sig…