7 papers · 1 filter
LibraSpec: Dynamic Diffusion-Based Speculative Decoding via Marginal-Gain-Driven Optimization
Zexun Lin, Yuan Feng, Junlin Lv +2
Speculative decoding accelerates large language model inference by drafting multiple tokens for parallel verification, with efficiency critically determined by the speculative leng…
Taming the Fragility of KV Cache Eviction in LLM Inference
Yuan Feng, Haoyu Guo, JunLin Lv +2
Large language models have revolutionized natural language processing, yet their deployment remains hampered by the substantial memory and runtime overhead of the transformer's Key…
CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspective
Yuan Feng, Junlin Lv, Haoyu Guo +3
Large language models have revolutionized natural language processing but face significant challenges of high storage and runtime costs, due to the transformer architecture's relia…
FRAG: A Flexible Modular Framework for Retrieval-Augmented Generation based on Knowledge Graphs
Zengyi Gao, Yukun Cao, Hairu Wang +4
To mitigate the hallucination and knowledge deficiency in large language models (LLMs), Knowledge Graph (KG)-based Retrieval-Augmented Generation (RAG) has shown promising potentia…
CritiPrefill: A Segment-wise Criticality-based Approach for Prefilling Acceleration in LLMs
Junlin Lv, Yuan Feng, Xike Xie +3
Large language models have achieved notable success across various domains, yet efficient inference is still limited by the quadratic computation complexity of the attention mechan…
GraphInsight: Unlocking Insights in Large Language Models for Graph Structure Understanding
Yukun Cao, Shuo Han, Zengyi Gao +3
Although Large Language Models (LLMs) have demonstrated potential in processing graphs, they struggle with comprehending graphical structure information through prompts of graph de…