4 papers · 1 filter
MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Any-Precision LLM
Dongwei Wang, Jinhee Kim, Seokho Han +8
Dynamic runtime latency and memory constraints necessitate flexible large language model (LLM) deployment, where an LLM can be inferred with various quantization precisions based o…
GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs
Jianing Deng, Song Wang, Dongwei Wang +4
Mixture-of-Experts Large Language Models (MoE-LLMs) achieve strong performance but incur substantial memory overhead due to massive expert parameters. Mixed-precision quantization…
MSQ: Memory-Efficient Bit Sparsification Quantization
Seokho Han, Seoyeon Yoon, Jinhee Kim +4
As deep neural networks (DNNs) see increased deployment on mobile and edge devices, optimizing model efficiency has become crucial. Mixed-precision quantization is widely favored,…
Taming Sensitive Weights : Noise Perturbation Fine-tuning for Robust LLM Quantization
Dongwei Wang, Huanrui Yang
Quantization is a critical step to enable efficient LLM serving under limited resource. However, previous research observes that certain weights in the LLM, known as outliers, are…