5 papers
HybridKV: Hybrid KV Cache Compression for Efficient Multimodal Large Language Model Inference
Bowen Zeng, Feiyang Ren, Jun Zhang +4
Multimodal Large Language Models (MLLMs) have advanced unified reasoning over text, images, and videos, but their inference is hindered by the rapid growth of key-value (KV) caches…
RedParrot: Accelerating NL-to-DSL for Business Analytics via Query Semantic Caching
Tong Wang, Yongqin Xu, Jianfeng Zhang +6
Recently, at Xiaohongshu, the rapid expansion of e-commerce and advertising demands real-time business analytics with high accuracy and low latency. To meet this demand, systems ty…
SVFusion: A CPU-GPU Co-Processing Architecture for Large-Scale Real-Time Vector Search
Yuchen Peng, Dingyu Yang, Zhongle Xie +4
Approximate Nearest Neighbor Search (ANNS) underpins modern applications such as information retrieval and recommendation. With the rapid growth of vector data, efficient indexing…
SafeLoad: Efficient Admission Control Framework for Identifying Memory-Overloading Queries in Cloud Data Warehouses
Yifan Wu, Yuhan Li, Zhenhua Wang +8
Memory overload is a common form of resource exhaustion in cloud data warehouses. When database queries fail due to memory overload, it not only wastes critical resources such as C…
TokenTiming: A Dynamic Alignment Method for Universal Speculative Decoding Model Pairs
Sibo Xiao, Jinyuan Fu, Zhongle Xie +1
Accelerating the inference of large language models (LLMs) has been a critical challenge in generative AI. Speculative decoding (SD) substantially improves LLM inference efficiency…