9 papers
LibraSpec: Dynamic Diffusion-Based Speculative Decoding via Marginal-Gain-Driven Optimization
Zexun Lin, Yuan Feng, Junlin Lv +2
Speculative decoding accelerates large language model inference by drafting multiple tokens for parallel verification, with efficiency critically determined by the speculative leng…
CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspective
Yuan Feng, Junlin Lv, Haoyu Guo +3
Large language models have revolutionized natural language processing but face significant challenges of high storage and runtime costs, due to the transformer architecture's relia…
Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference
Yuan Feng, Junlin Lv, Yukun Cao +2
Large Language Models have excelled in various domains but face efficiency challenges due to the growing Key-Value (KV) cache required for long-sequence inference. Recent efforts a…
Taming the Fragility of KV Cache Eviction in LLM Inference
Yuan Feng, Haoyu Guo, JunLin Lv +2
Large language models have revolutionized natural language processing, yet their deployment remains hampered by the substantial memory and runtime overhead of the transformer's Key…
SkewRoute: Training-Free LLM Routing for Knowledge Graph Retrieval-Augmented Generation via Score Skewness of Retrieved Context
Hairu Wang, Yuan Feng, Yukun Cao +2
Large language models excel at many tasks but often incur high inference costs during deployment. To mitigate hallucination, many systems use a knowledge graph to enhance retrieval…
Path Pooling: Training-Free Structure Enhancement for Efficient Knowledge Graph Retrieval-Augmented Generation
Hairu Wang, Yuan Feng, Xike Xie +1
Although Large Language Models achieve strong success in many tasks, they still suffer from hallucinations and knowledge deficiencies in real-world applications. Many knowledge gra…