Which Decisions Low-Bit Quantization Breaks, and How to Predict Them

Zekun Wu, Swati Dhiman, Adriano Koshiyama

Quantization is known to hurt below four bits, but nobody can say which of a model's decisions will change at a given bit-width. This matters most where a model acts rather than an…