10 papers
CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspective
Yuan Feng, Junlin Lv, Haoyu Guo +3
Large language models have revolutionized natural language processing but face significant challenges of high storage and runtime costs, due to the transformer architecture's relia…
Stem: Rethinking Causal Information Flow in Sparse Attention
Lin Niu, Xin Luo, Linchuan Xie +4
The quadratic computational complexity of self-attention remains a fundamental bottleneck for scaling Large Language Models (LLMs) to long contexts, particularly during the pre-fil…
See or Say Graphs: Agent-Driven Scalable Graph Structure Understanding with Vision-Language Models
Shuo Han, Yukun Cao, Zezhong Ding +3
Vision-language models (VLMs) have shown promise in graph structure understanding, but remain limited by input-token constraints, facing scalability bottlenecks and lacking effecti…
Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference
Yuan Feng, Junlin Lv, Yukun Cao +2
Large Language Models have excelled in various domains but face efficiency challenges due to the growing Key-Value (KV) cache required for long-sequence inference. Recent efforts a…
Taming the Fragility of KV Cache Eviction in LLM Inference
Yuan Feng, Haoyu Guo, JunLin Lv +2
Large language models have revolutionized natural language processing, yet their deployment remains hampered by the substantial memory and runtime overhead of the transformer's Key…
LEGO-GraphRAG: Modularizing Graph-based Retrieval-Augmented Generation for Design Space Exploration
Yukun Cao, Zengyi Gao, Zhiyang Li +3
GraphRAG integrates (knowledge) graphs with large language models (LLMs) to improve reasoning accuracy and contextual relevance. Despite its promising applications and strong relev…