7 papers
Making Every Verified Token Count: Adaptive Verification for MoE Speculative Decoding
Lehan Pan, Ziyang Tao, Xiao Wang +3
Tree-based speculative decoding accelerates autoregressive generation by verifying multiple draft candidates in parallel, but this advantage weakens for sparse Mixture-of-Experts (…
DepCap: Adaptive Block-Wise Parallel Decoding for Efficient Diffusion LM Inference
Xiang Xia, Wuyang Zhang, Jiazheng Liu +2
Diffusion language models (DLMs) have emerged as a promising alternative to autoregressive language generation due to their potential for parallel decoding and global refinement of…
Understanding the Role of Hallucination in Reinforcement Post-Training of Multimodal Reasoning Models
Gengwei Zhang, Jie Peng, Zhen Tan +6
The recent success of reinforcement learning (RL) in large reasoning models has inspired the growing adoption of RL for post-training Multimodal Large Language Models (MLLMs) to en…
T-MAN: Enabling End-to-End Low-Bit LLM Inference on NPUs via Unified Table Lookup
Jianyu Wei, Qingtao Li, Shijie Cao +5
Large language models (LLMs) are increasingly deployed on customer devices. To support them, current devices are adopting SoCs (System on Chip) with NPUs (Neural Processing Unit) i…
GRACE-MoE: Grouping and Replication with Locality-Aware Routing for Efficient Distributed MoE Inference
Yu Han, Lehan Pan, Jie Peng +4
Sparse Mixture of Experts (SMoE) enables scalable parameter growth in large language models (LLMs) by selectively activating a subset of experts, and its large parameter count nece…
LightDefense: A Lightweight Uncertainty-Driven Defense against Jailbreaks via Shifted Token Distribution
Zhuoran Yang, Yanyong Zhang
Large Language Models (LLMs) face threats from jailbreak prompts. Existing methods for defending against jailbreak attacks are primarily based on auxiliary models. These strategies…
Harnessing Your DRAM and SSD for Sustainable and Accessible LLM Inference with Mixed-Precision and Multi-level Caching
Jie Peng, Zhang Cao, Huaizhi Qu +5
Although Large Language Models (LLMs) have demonstrated remarkable capabilities, their massive parameter counts and associated extensive computing make LLMs' deployment the main pa…