Enabling Dynamic Sparsity in Quantized LLM Inference

Rongxiang Wang, Kangyuan Shu, Felix Xiaozhu Lin

Deploying large language models (LLMs) on end-user devices is gaining importance due to benefits in responsiveness, privacy, and operational cost. Yet the limited memory and comput…