RSQ: Learning from Important Tokens Leads to Better Quantized LLMs

Yi-Lin Sung, Prateek Yadav, Jialu Li +2

Layer-wise quantization is a key technique for efficiently compressing large models without expensive retraining. Previous methods typically quantize the weights of each layer by "…