Which Decisions Low-Bit Quantization Breaks, and How to Predict Them

Zekun Wu, Swati Dhiman, Adriano Koshiyama

Quantization saves memory by storing model weights with fewer bits. It can also change model decisions, such as whether to call a tool or which option to choose from a finite set.…