any4: Learned 4-bit Numeric Representation for LLMs

Mostafa Elhoushi, Jeff Johnson

We present any4, a learned 4-bit weight quantization solution for large language models (LLMs) providing arbitrary numeric representations without requiring pre-processing of weigh…