GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance

Jinuk Kim, Marwa El Halabi, Wonpyo Park +5

Post-training quantization is a key technique for reducing the memory and inference latency of large language models by quantizing weights and activations without requiring retrain…