4 papers
Llama-Mobile: Efficient 2.7-Bit Quantization of VLMs
Luka Ribar, Jeevan Bhoot, Douglas Orr
Deploying vision-language models (VLMs) on mobile devices is challenging due to their significant memory and compute requirements. We present a framework for quantizing VLMs for ef…
Studying quantization trade-offs for efficient inference deployment in machine translation
Jim Zhao, Sohir Maskey, Koen Oostermeijer +2
Deploying large language models in realistic server environments poses challenges, as the system needs to provide high-quality responses with low latency. Quantization is a common…
1-Bit Wonder: Improving QAT Performance in the Low-Bit Regime through K-Means Quantization
Sohir Maskey, Constantin Eichenberg, Johannes Messner +1
Quantization-aware training (QAT) is an effective method to drastically reduce the memory footprint of LLMs while keeping performance degradation at an acceptable level. However, t…
Approximate Top- for Increased Parallelism
Oscar Key, Luka Ribar, Alberto Cattaneo +2
We present an evaluation of bucketed approximate top- algorithms. Computing top- exactly suffers from limited parallelism, because the largest values must be aggregated a…