Twice as Fast, Until It Isn't: Quantized Serving of Mid-Size Open LLMs

Smit Kadvani, Kriti Faujdar, Spurthi Tallam

A team serving a 9B-20B open-weight language model on vLLM or SGLang must choose a quantization recipe, and needs two numbers: the speedup over BF16 and the accuracy given up. We a…