End-to-End On-Device Quantization-Aware Training for LLMs at Inference Cost

Qitao Tan, Xiaoying Song, Ningxi Cheng +11

Quantization reduces the deployment cost of large language models (LLMs). However, post-training quantization (PTQ) cannot fine-tune model parameters and often degrades accuracy in…