8 papers
POP: Prefill-Only Pruning for Efficient Large Model Inference
Junhui He, Zhihui Fu, Jun Wang +1
Large Language Models (LLMs) and Vision-Language Models (VLMs) have demonstrated remarkable capabilities. However, their deployment is hindered by significant computational costs.…
Knowledge-Driven Multi-Turn Jailbreaking on Large Language Models
Songze Li, Ruishi He, Xiaojun Jia +2
Large Language Models (LLMs) face a significant threat from multi-turn jailbreak attacks, where adversaries progressively steer conversations to elicit harmful outputs. However, th…
When, What, and How: Rethinking Retrieval-Enhanced Speculative Decoding
Min Fang, Zhihui Fu, Qibin Zhao +1
Speculative decoding (SD) has emerged as an effective technique to accelerate large language model (LLM) inference without compromising output quality. However, the achievable spee…
DiffuSpec: Unlocking Diffusion Language Models for Speculative Decoding
Guanghao Li, Zhihui Fu, Min Fang +4
As large language models (LLMs) scale up, accuracy improves, but the autoregressive (AR) nature of decoding increases latency since each token requires a serial forward pass. Specu…
GreedyPrune: Retenting Critical Visual Token Set for Large Vision Language Models
Ruiguang Pei, Weiqing Sun, Zhihui Fu +1
Although Large Vision Language Models (LVLMs) have demonstrated remarkable performance in image understanding tasks, their computational efficiency remains a significant challenge,…
Accelerating Prefilling for Long-Context LLMs via Sparse Pattern Sharing
Dan Peng, Zhihui Fu, Zewen Ye +2
Sparse attention methods exploit the inherent sparsity in attention to speed up the prefilling phase of long-context inference, mitigating the quadratic complexity of full attentio…
SimDC: A High-Fidelity Device Simulation Platform for Device-Cloud Collaborative Computing
Ruiguang Pei, Junjie Wu, Dan Peng +4
The advent of edge intelligence and escalating concerns for data privacy protection have sparked a surge of interest in device-cloud collaborative computing. Large-scale device dep…
PocketLLM: Enabling On-Device Fine-Tuning for Personalized LLMs
Dan Peng, Zhihui Fu, Jun Wang
Recent advancements in large language models (LLMs) have indeed showcased their impressive capabilities. On mobile devices, the wealth of valuable, non-public data generated daily…