9 papers
LiveServe: Interaction-Aware Serving for Real-Time Omni-Modal LLMs
Xiangyu Zhi, Peiqi Yin, Sheng Guan +3
Realtime omni-modal LMs support speech-centric conversations where users stream inputs, hear generated audio, and interrupt freely. Existing Omni-LM serving systems still rely on t…
Search-R3: Unifying Reasoning and Embedding in Large Language Models
Yuntao Gui, James Cheng
Despite their remarkable natural language understanding capabilities, Large Language Models (LLMs) have been underutilized for retrieval tasks. We present Search-R3, a novel framew…
vLLM-Omni: Fully Disaggregated Serving for Any-to-Any Multimodal Models
Peiqi Yin, Jiangyun Zhu, Han Gao +13
Any-to-any multimodal models that jointly handle text, images, video, and audio represent a significant advance in multimodal AI. However, their complex architectures (typically co…
SAQ: Pushing the Limits of Vector Quantization through Code Adjustment and Dimension Segmentation
Hui Li, Shiyuan Deng, Xiao Yan +2
Approximate Nearest Neighbor Search (ANNS) plays a critical role in applications such as search engines, recommender systems, and RAG for LLMs. Vector quantization (VQ), a crucial…
SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Serving
Qihui Zhou, Peiqi Yin, Pengfei Zuo +1
Serving long-context LLMs is costly because attention computation grows linearly with context length. Dynamic sparse attention algorithms (DSAs) mitigate this by attending only to…
Gorgeous: Revisiting the Data Layout for Disk-Resident High-Dimensional Vector Search
Peiqi Yin, Xiao Yan, Qihui Zhou +6
Similarity-based vector search underpins many important applications, but a key challenge is processing massive vector datasets (e.g., in TBs). To reduce costs, some systems utiliz…