Pushing the Envelope of LLM Inference with Ultra-Low-Bit Quantized Models

Evangelos Georganas, Dhiraj Kalamkar, Alexander Heinecke +1

The advent of ultra-low-bit LLM models, approaching the perplexity and task accuracy of their full precision counterparts, is ushering in a new era of LLM inference. While these ad…