9 papers
Beyond the Target: From Imitation to Collaboration in Speculative Decoding
Jinze Li, Yixing Xu, Guanchen Li +7
Speculative decoding (SPD) accelerates large language model (LLM) inference by letting a smaller draft model propose multiple future tokens that are verified in parallel by a large…
Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match
Jinze Li, Yixing Xu, Guanchen Li +6
Large language models (LLMs) achieve strong performance across diverse tasks but suffer from high inference latency due to their autoregressive generation. Speculative Decoding (SP…
Learnable Permutation for Structured Sparsity on Transformer Models
Zekai Li, Ji Liu, Guanchen Li +5
Structured sparsity has emerged as a popular model pruning technique, widely adopted in various architectures, including CNNs, Transformer models, and especially large language mod…
Dual LoRA: Enhancing LoRA with Magnitude and Direction Updates
Yixing Xu, Chao Li, Xuanwu Yin +4
Low-rank adaptation (LoRA) is one of the most popular methods among parameter-efficient fine-tuning (PEFT) methods to adapt pre-trained large language models (LLMs) to specific dow…
SparK: Query-Aware Unstructured Sparsity with Recoverable KV Cache Channel Pruning
Huanxuan Liao, Yixing Xu, Shizhu He +6
Long-context inference in large language models (LLMs) is increasingly constrained by the KV cache bottleneck: memory usage grows linearly with sequence length, while attention com…
Týr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization
Guanchen Li, Yixing Xu, Zeping Li +4
Structural pruning enhances hardware-agnostic inference efficiency for large language models (LLMs) yet often fails to maintain comparable performance. Local pruning performs effic…