GPTVQ: The Blessing of Dimensionality for LLM Quantization

Mart van Baalen, Andrey Kuzmin, Ivan Koryakovskiy +6

In this work we show that the size versus accuracy trade-off of neural network quantization can be significantly improved by increasing the quantization dimensionality. We propose…