OnlineQAT: On-Policy Distillation for Ultra-Low-Bit Large Language Models

Wenjun Wang, Heng Li, Yanggan Gu +1

Quantization-aware training (QAT) can recover much of the accuracy lost when large language models are compressed below four bits. Existing re- covery stages, however, are commonly…