7 papers
dMX: Differentiable Mixed-Precision Assignment for Low-Precision Floating-Point Formats
Giuseppe Franco, Ian Colbert, Pablo Monteagudo-Lago +2
Quantizing large language models (LLMs) to low-precision floating-point representations is central to efficient deployment, yet applying a single bit-width uniformly across all lay…
Signed Symmetric Quantization for Few-Bit Integers
Ian Colbert, Eashan Dash, Pablo Monteagudo-Lago +5
The signed integer alphabet contains one more negative representable value than positive. Yet, by convention, the standard symmetric integer quantizer fixes its scale to be strictl…
Optimal Post-Training Quantization Scales and Where to Find Them
Juan Amboage, Pablo Monteagudo-Lago, Ian Colbert +2
Post-training quantization (PTQ) compresses large language models by mapping weights to low-bit representations. While much work has focused on how weights are rounded onto the qua…
Pushing the Limits of Block Rotations in Post-Training Quantization
Sai Sanjeet, Ian Colbert, Pablo Monteagudo-Lago +3
Recent post-training quantization (PTQ) methods have adopted block rotations to diffuse outliers prior to rounding. While this reduces the overhead of online full-vector rotations,…
Improving Quantization with Post-Training Model Expansion
Giuseppe Franco, Pablo Monteagudo-Lago, Ian Colbert +2
The size of a model has been a strong predictor of its quality, as well as its cost. As such, the trade-off between model cost and quality has been well-studied. Post-training opti…
Accumulator-Aware Post-Training Quantization for Large Language Models
Ian Colbert, Giuseppe Franco, Fabian Grob +2
When quantizing weights and activations to increasingly narrower representations, the cost of additions begins to dominate that of multiplications in multiply-accumulate (MAC) unit…
Shedding the Bits: Pushing the Boundaries of Quantization with Minifloats on FPGAs
Shivam Aggarwal, Hans Jakob Damsgaard, Alessandro Pappalardo +4
Post-training quantization (PTQ) is a powerful technique for model compression, reducing the numerical precision in neural networks without additional training overhead. Recent wor…