Why Does Post-Training Quantization Work?

Yuxiang Chen, Michael Beyer, Jun Zhu +1

Post-training quantization compresses large language models (LLMs) by storing their weights at reduced precision, and each quantized weight introduces an error into the hidden stat…