5 papers · 1 filter
When, What, and How: Rethinking Retrieval-Enhanced Speculative Decoding
Min Fang, Zhihui Fu, Qibin Zhao +1
Speculative decoding (SD) has emerged as an effective technique to accelerate large language model (LLM) inference without compromising output quality. However, the achievable spee…
DiffuSpec: Unlocking Diffusion Language Models for Speculative Decoding
Guanghao Li, Zhihui Fu, Min Fang +4
As large language models (LLMs) scale up, accuracy improves, but the autoregressive (AR) nature of decoding increases latency since each token requires a serial forward pass. Specu…
GreedyPrune: Retenting Critical Visual Token Set for Large Vision Language Models
Ruiguang Pei, Weiqing Sun, Zhihui Fu +1
Although Large Vision Language Models (LVLMs) have demonstrated remarkable performance in image understanding tasks, their computational efficiency remains a significant challenge,…
Accelerating Prefilling for Long-Context LLMs via Sparse Pattern Sharing
Dan Peng, Zhihui Fu, Zewen Ye +2
Sparse attention methods exploit the inherent sparsity in attention to speed up the prefilling phase of long-context inference, mitigating the quadratic complexity of full attentio…
SimDC: A High-Fidelity Device Simulation Platform for Device-Cloud Collaborative Computing
Ruiguang Pei, Junjie Wu, Dan Peng +4
The advent of edge intelligence and escalating concerns for data privacy protection have sparked a surge of interest in device-cloud collaborative computing. Large-scale device dep…