What Makes Quantization for Large Language Models Hard? An Empirical Study from the Lens of Perturbation

Zhuocheng Gong, Jiahao Liu, Jingang Wang +3

Quantization has emerged as a promising technique for improving the memory and computational efficiency of large language models (LLMs). Though the trade-off between performance an…