QIGen: Generating Efficient Kernels for Quantized Inference on Large Language Models

Tommaso Pegolotti, Elias Frantar, Dan Alistarh +1

We present ongoing work on a new automatic code generation approach for supporting quantized generative inference on LLMs such as LLaMA or OPT on off-the-shelf CPUs. Our approach i…