Total Variation Distance Estimation in Autoregressive Models

Abstract

Modern LLM deployments use a number of implementation choices and inference optimizations (e.g., batching, custom kernels, and quantization) on top of fixed weights, so two engines serving "the same model" can produce meaningfully different distributions. We study the problem of estimating the total variation (TV) distance between two length-$n$ autoregressive distributions to additive error $\epsilon$, under three access models. (1) Under sample access, we use $\tilde{O}(n^{2}K/{\epsilon }^2)$ queries, where $K$ is the maximum support of the next-token distribution. This improves upon the $\tilde{O}(n^{3}m/{\epsilon }^5)$-query estimator of Meel et al. (2025), where $m\ge K$ is the total size of the token alphabet. (2) Under logit access, we use $O(n/{\epsilon }^2)$ queries, and this is tight. (3) Under noisy logit access, we smoothly interpolate between the above two guarantees: if probability values are given to relative error $\ddot{I}ƒ$, we use $\tilde{O}((n+n^2\ddot{I}{ƒ}^2)/{\epsilon }^2)$ queries. We complement our theoretical results with an empirical evaluation of our algorithms, for example measuring the distance between SGLang and vLLM serving identical weights. Our experiments highlight the robustness and practicality of estimating the total variation distance, which remains estimable where the KL divergence is infinite. Our code is available at https://github.com/XunZhiyang/llm-tv-estimation.

39 pages, 11 figures, code is available at https://github.com/XunZhiyang/llm-tv-estimation