8 papers
Learnable Permutation for Structured Sparsity on Transformer Models
Zekai Li, Ji Liu, Guanchen Li +5
Structured sparsity has emerged as a popular model pruning technique, widely adopted in various architectures, including CNNs, Transformer models, and especially large language mod…
Dual LoRA: Enhancing LoRA with Magnitude and Direction Updates
Yixing Xu, Chao Li, Xuanwu Yin +4
Low-rank adaptation (LoRA) is one of the most popular methods among parameter-efficient fine-tuning (PEFT) methods to adapt pre-trained large language models (LLMs) to specific dow…
Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match
Jinze Li, Yixing Xu, Guanchen Li +6
Large language models (LLMs) achieve strong performance across diverse tasks but suffer from high inference latency due to their autoregressive generation. Speculative Decoding (SP…
SpecVLM: Fast Speculative Decoding in Vision-Language Models
Haiduo Huang, Fuwei Yang, Zhenhua Liu +4
Speculative decoding is a powerful way to accelerate autoregressive large language models (LLMs), but directly porting it to vision-language models (VLMs) faces unique systems cons…
SparK: Query-Aware Unstructured Sparsity with Recoverable KV Cache Channel Pruning
Huanxuan Liao, Yixing Xu, Shizhu He +6
Long-context inference in large language models (LLMs) is increasingly constrained by the KV cache bottleneck: memory usage grows linearly with sequence length, while attention com…
Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding
Jinze Li, Yixing Xu, Haiduo Huang +4
Speculative decoding (SPD) aims to accelerate the auto-regressive token generation process of a target Large Language Model (LLM). Some approaches employ a draft model with multipl…