MicroMix: Efficient Mixed-Precision Quantization with Microscaling Formats for Large Language Models

Wenyuan Liu, Haoqian Meng, Yilun Luo +3

Quantization significantly accelerates inference in large language models (LLMs) by replacing original high-precision matrices with low-precision counterparts. Recent advances in w…