The Impact of Quantization on Retrieval-Augmented Generation: An Analysis of Small LLMs

Mert Yazan, Suzan Verberne, Frederik Situmeang

Post-training quantization reduces the computational demand of Large Language Models (LLMs) but can weaken some of their capabilities. Since LLM abilities emerge with scale, smalle…