6 papers
Accelerate Speculative Decoding with Sparse Computation in Verification
Jikai Wang, Jianchao Tan, Yuxuan Hu +6
Speculative decoding accelerates autoregressive language model inference by verifying multiple draft tokens in parallel. However, the verification stage often becomes the dominant…
Alignment-Augmented Speculative Decoding with Alignment Sampling and Conditional Verification
Jikai Wang, Zhenxu Tian, Juntao Li +5
Recent works have revealed the great potential of speculative decoding in accelerating the autoregressive generation process of large language models. The success of these methods…
CaliDrop: KV Cache Compression with Calibration
Yi Su, Quantong Qiu, Yuechi Zhou +6
Large Language Models (LLMs) require substantial computational resources during generation. While the Key-Value (KV) cache significantly accelerates this process by storing attenti…
LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation framework
Zecheng Tang, Haitian Wang, Quantong Qiu +5
Long-context processing has become a fundamental capability for large language models~(LLMs). To assess model's long-context performance, numerous long-context evaluation benchmark…
Beware of Calibration Data for Pruning Large Language Models
Yixin Ji, Yang Xiang, Juntao Li +5
As large language models (LLMs) are widely applied across various fields, model compression has become increasingly crucial for reducing costs and improving inference efficiency. P…
Accurate KV Cache Quantization with Outlier Tokens Tracing
Yi Su, Yuechi Zhou, Quantong Qiu +6
The impressive capabilities of Large Language Models (LLMs) come at the cost of substantial computational resources during deployment. While KV Cache can significantly reduce recom…