Integer or Floating Point? New Outlooks for Low-Bit Quantization on Large Language Models

Yijia Zhang, Lingran Zhao, Shijie Cao +6

Efficient deployment of large language models (LLMs) necessitates low-bit quantization to minimize model size and inference cost. While low-bit integer formats (e.g., INT8/INT4) ha…