6 papers
: Attention-Aware Accurate KV Cache Fusion for Fast Large Language Model Serving
Yuechi Zhou, Yi Su, Jianxin Zhang +5
Large language models (LLMs) have demonstrated strong capabilities in processing long contexts, enabling them to tackle tasks involving long textual inputs such as multi-turn conve…
Alignment-Augmented Speculative Decoding with Alignment Sampling and Conditional Verification
Jikai Wang, Zhenxu Tian, Juntao Li +5
Recent works have revealed the great potential of speculative decoding in accelerating the autoregressive generation process of large language models. The success of these methods…
Beware of Calibration Data for Pruning Large Language Models
Yixin Ji, Yang Xiang, Juntao Li +5
As large language models (LLMs) are widely applied across various fields, model compression has become increasingly crucial for reducing costs and improving inference efficiency. P…
Accurate KV Cache Quantization with Outlier Tokens Tracing
Yi Su, Yuechi Zhou, Quantong Qiu +6
The impressive capabilities of Large Language Models (LLMs) come at the cost of substantial computational resources during deployment. While KV Cache can significantly reduce recom…
Taming the Titans: A Survey of Efficient LLM Inference Serving
Ranran Zhen, Juntao Li, Yixin Ji +7
Large Language Models (LLMs) for Generative AI have achieved remarkable progress, evolving into sophisticated and versatile tools widely adopted across various domains and applicat…
OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure
Jikai Wang, Yi Su, Juntao Li +5
Autoregressive language models demonstrate excellent performance in various scenarios. However, the inference efficiency is limited by its one-step-one-word generation mode, which…