Studying quantization trade-offs for efficient inference deployment in machine translation

Jim Zhao, Sohir Maskey, Koen Oostermeijer +2

Deploying large language models in realistic server environments poses challenges, as the system needs to provide high-quality responses with low latency. Quantization is a common…