VQ-LLM: High-performance Code Generation for Vector Quantization Augmented LLM Inference

Zihan Liu, Xinhao Luo, Junxian Guo +11

In this work, we design and implement VQ-LLM, an efficient fused Vector Quantization (VQ) kernel generation framework. We first introduce a software abstraction called codebook cac…