Optimal Scalar Quantization for Matrix Multiplication: Closed-Form Density and Phase Transition

Abstract

We study entrywise scalar quantization of two matrices prior to multiplication. Given $A\in R^{m\times k}$ and $B\in R^{k\times n}$, we quantize entries of $A$ and $B$ independently using scalar quantizers with $K_X$ and $K_Y$ levels per entry, and form $\hat{C}=\hat{A}\hat{B}$. The objective is to minimize the matrix multiplication mean-squared error (MSE) $E[\Vert AB-\hat{A}\hat{B}{\Vert }_F^2]$ under a pair-i.i.d.\ inner-product model. In the high-resolution regime $K_X,K_Y\to \infty$, we derive a sharp $K^{-2}$ asymptotic expansion for $\mathcal{E}$, identify the exact optimal leading constants, and characterize asymptotically optimal quantization center densities in terms of conditional second moments. We then specialize to correlated Gaussian multiplicative pairs, obtaining a closed-form optimal point density \[ λ^\star(u)\ \propto\ \exp\!\left(-\frac{u^2}{6}\right)\bigl((1-ρ^2)+ρ^2u^2\bigr)^{1/3}, \qquad u=\frac{x}{σ_X}, \] with the same form for $y/\ddot{I}{ƒ}_Y$, and prove a correlation-driven phase transition: the density is unimodal at the origin for $|\ddot{I}|\le 1/\sqrt{3}$ and becomes bimodal for $|\ddot{I}|>1/\sqrt{3}$ with peaks at $u_{\mathrm{peak}}=\pm \sqrt{3-1/\ddot{I}{}^2}$. We show our method's applicability in synthetic experiments such as matrix multiplication quantization and least squares optimization, as well as quantization of large language model key and query activations.