MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference

Simla Burcu Harma, Danila Mishin, Zhengyuan Su +7

4-bit quantization enables efficient LLM inference, but suffers from significant accuracy degradation due to outliers. Prior work addresses this problem via data rotation or mixed-…