4 papers
Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Serving
Rui Li, Zhaoning Zhang, Libo Zhang +3
Speculative decoding (SD) accelerates LLM inference by verifying draft tokens in parallel. However, this method presents a critical trade-off: it improves throughput in low-load, m…
Sparrow: Text-Anchored Window Attention with Visual-Semantic Glimpsing for Speculative Decoding in Video LLMs
Libo Zhang, Zhaoning Zhang, Wangyang Hong +2
Although speculative decoding is widely used to accelerate Vision-Language Models (VLMs) inference, it faces severe performance collapse when applied to Video Large Language Models…
Rhea: Role-aware Heuristic Episodic Attention for Conversational LLMs
Wanyang Hong, Zhaoning Zhang, Yi Chen +5
Large Language Models (LLMs) have achieved remarkable performance on single-turn tasks, yet their effectiveness deteriorates in multi-turn conversations. We define this phenomenon…
Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inference
Libo Zhang, Zhaoning Zhang, Baizhou Xu +4
With the continuous advancement in the performance of large language models (LLMs), their demand for computational resources and memory has significantly increased, which poses maj…