9 papers
Unlocking Parallelism in Autoregressive Language Models via Speculative Decoding with Progressive Tree Drafting
Zipeng Gao, Zhi Zheng, Qingrong Xia +5
Speculative decoding has significantly accelerated Large Language Model (LLM) inference by alleviating memory-bound bottlenecks. However, traditional speculative decoding typically…
: Attention-Aware Accurate KV Cache Fusion for Fast Large Language Model Serving
Yuechi Zhou, Yi Su, Jianxin Zhang +5
Large language models (LLMs) have demonstrated strong capabilities in processing long contexts, enabling them to tackle tasks involving long textual inputs such as multi-turn conve…
CaliDrop: KV Cache Compression with Calibration
Yi Su, Quantong Qiu, Yuechi Zhou +6
Large Language Models (LLMs) require substantial computational resources during generation. While the Key-Value (KV) cache significantly accelerates this process by storing attenti…
Accurate KV Cache Quantization with Outlier Tokens Tracing
Yi Su, Yuechi Zhou, Quantong Qiu +6
The impressive capabilities of Large Language Models (LLMs) come at the cost of substantial computational resources during deployment. While KV Cache can significantly reduce recom…
Alignment-Augmented Speculative Decoding with Alignment Sampling and Conditional Verification
Jikai Wang, Zhenxu Tian, Juntao Li +5
Recent works have revealed the great potential of speculative decoding in accelerating the autoregressive generation process of large language models. The success of these methods…
Taming the Titans: A Survey of Efficient LLM Inference Serving
Ranran Zhen, Juntao Li, Yixin Ji +7
Large Language Models (LLMs) for Generative AI have achieved remarkable progress, evolving into sophisticated and versatile tools widely adopted across various domains and applicat…
Beware of Calibration Data for Pruning Large Language Models
Yixin Ji, Yang Xiang, Juntao Li +5
As large language models (LLMs) are widely applied across various fields, model compression has become increasingly crucial for reducing costs and improving inference efficiency. P…
OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure
Jikai Wang, Yi Su, Juntao Li +5
Autoregressive language models demonstrate excellent performance in various scenarios. However, the inference efficiency is limited by its one-step-one-word generation mode, which…
A General and Flexible Multi-concept Parsing Framework for Multilingual Semantic Matching
Dong Yao
Sentence semantic matching is a research hotspot in natural language processing, which is considerably significant in various key scenarios, such as community question answering, s…