4 papers
BELT : BEhavior-Aligned Low-Rank KV Cache Compression with Learned Top-k Rank Allocation
Yaoyang Liu, Shuwei Shen, Jincheng Tang +5
Long-context inference makes the key-value (KV) cache a major memory bottleneck for large language models. We propose BELT, a low-rank KV cache compression framework. Its key-side…
MVR-cache: Optimizing Semantic Caching via Multi-Vector Retrieval and Learned Prompt Segmentation
Ali Noshad, Zishan Zheng, Yinjun Wu
To reduce LLM costs and latency, semantic caching systems must accurately identify when a new prompt matches a cached one. Current methods often rely on simplistic similarity measu…
MIRAGE: Runtime Scheduling for Multi-Vector Image Retrieval with Hierarchical Decomposition
Maoliang Li, Ke Li, Yaoyang Liu +5
To effectively leverage user-specific data, retrieval augmented generation (RAG) is employed in multimodal large language model (MLLM) applications. However, conventional retrieval…
POQD: Performance-Oriented Query Decomposer for Multi-vector retrieval
Yaoyang Liu, Junlin Li, Yinjun Wu +1
Although Multi-Vector Retrieval (MVR) has achieved the state of the art on many information retrieval (IR) tasks, its performance highly depends on how to decompose queries into sm…