From the 1 of 5 linked papers with an AI index.
5 papers
AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding
Hong Liu, Rui Cen, Junhan Shi +10
Speculative decoding accelerates large language model inference without changing the target distribution, but no single drafting structure performs best across real-world workloads…
D-cut: Adaptive Verification Depth Pruning for Batched Speculative Decoding
Tianyu Liu, Yuhao Shen, Rui Cen +7
The paper introduces D-Cut, an adaptive method that prunes draft tokens across a batch to focus verification on the most promising tokens, improving the speed of speculative decodi…
SOAR: Scale Optimization for Accurate Reconstruction in NVFP4 Quantization
Chengzhu Bao, Xianglong Yan, Zhiteng Li +3
NVFP4 has recently emerged as an efficient 4-bit microscaling format for large language models (LLMs), offering superior numerical fidelity with native hardware support. However, e…
VEQ: Modality-Adaptive Quantization for MoE Vision-Language Models
Guangshuo Qin, Zhiteng Li, Zheng Chen +3
Mixture-of-Experts(MoE) Vision-Language Models (VLMs) offer remarkable performance but incur prohibitive memory and computational costs, making compression essential. Post-Training…
SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deployment
Yixin Song, Zhenliang Xue, Dongliang Wei +11
While frontier large language models (LLMs) continue to push capability boundaries, their deployment remains confined to GPU-powered cloud infrastructure. We challenge this paradig…