5 papers
AdaFlash: Adaptive Speculative Decoding via On-Policy Distilled Diffusion Drafters
Yu-Yang Qian, Hao-Cong Wu, Chen Chen +4
Speculative decoding, in which a lightweight draft model first generates a draft sequence that is then verified in parallel by the target model, has become a prevalent paradigm for…
JetSpec: Breaking the Scaling Ceiling of Speculative Decoding with Parallel Tree Drafting
Lanxiang Hu, Zhaoxiang Feng, Yulun Wu +9
Speculative decoding (SD) accelerates autoregressive Large Language Models (LLMs) by drafting multiple tokens and verifying them in parallel, but it faces a scaling limitation: inc…
Dynamic Chunking for Diffusion Language Models
Yichen Zhu, Xiaoming Shi, Peng Zhao +3
Block discrete diffusion language models factorize a sequence autoregressively over fixed-size positional blocks, decoupling within-block parallel denoising from across-block condi…
When Drafts Evolve: Speculative Decoding Meets Online Learning
Yu-Yang Qian, Hao-Cong Wu, Yichao Fu +2
Speculative decoding has emerged as a widely adopted paradigm for accelerating large language model inference, where a lightweight draft model rapidly generates candidate tokens th…
Choose Your Model Size: Any Compression of Large Language Models Without Re-Computation
Martin Genzel, Patrick Putzky, Pengfei Zhao +5
The adoption of Foundation Models in resource-constrained environments remains challenging due to their large size and inference costs. A promising way to overcome these limitation…