4 papers
LUMEN: Coordinated Failure Recovery for Distributed LLM Serving
Zhang Cao, Shujie Han, Juncheng Zhang +3
Modern large language model (LLM) serving clusters distribute inference requests across multiple worker processes on different GPUs, but failures are prevalent at scale. When a wor…
Leveraging I/O Stalls for Efficient Scheduling in ANNS
Juncheng Zhang, Yuanming Ren, Yongkun Li +1
Disk-based graph indexes for approximate nearest neighbor search (ANNS) must serve latency-sensitive queries and throughput-demanding updates concurrently. We observe that over 40%…
Decoupling Vector Data and Index Storage for Space Efficiency
Yuanming Ren, Juncheng Zhang, Yanjing Ren +3
Managing large-scale vector datasets with disk-resident graph approximate nearest neighbor search (ANNS) systems incurs substantial storage overhead due to the co-location of vecto…
CentroidKV: Efficient Long-Context LLM Inference via KV Cache Clustering
Jie Hu, Shengnan Wang, Yutong He +8
Large language models (LLMs) with extended context windows have become increasingly prevalent for tackling complex tasks. However, the substantial Key-Value (KV) cache required for…