QQQ: Quality Quattuor-Bit Quantization for Large Language Models

Ying Zhang, Peng Zhang, Mincong Huang +7

Quantization is a proven effective method for compressing large language models. Although popular techniques like W8A8 and W4A16 effectively maintain model performance, they often…