LLM-FP4: 4-Bit Floating-Point Quantized Transformers

Shih-yang Liu, Zechun Liu, Xijie Huang +2

We propose LLM-FP4 for quantizing both weights and activations in large language models (LLMs) down to 4-bit floating-point values, in a post-training manner. Existing post-trainin…