How Much Can Repeated Running Buy? A Measured Variance Ceiling for LLM Code Evaluation

Abstract

A benchmark score is a measurement whose uncertainty has two sources: an aleatoric component from stochastic decoding and an epistemic component from the finite problem set. Using 16 models on 99 recent LeetCode-style problems with two prompts and five runs each (15,840 judged runs in a fully crossed design), we estimate both components and ask a design question: how much can repeated running buy? First, the pooled-binomial interval that repeated-run protocols invite answers neither question of interest---it is about $2\times$ too narrow for generalisation to new problems and $2\times$ too wide for the fixed problem set---and under correctly clustered intervals, 12 of 15 adjacent leaderboard pairs are statistically unresolved. Second, the variance ceiling $1+{\mu }_w/{\sigma }_b^2$, the largest factor by which repetition can reduce score variance, has median 1.34 across models (range 1.05--1.71), so repetition can narrow confidence intervals by at most 23% on this benchmark, and by 14% for the median model. Third, once the cost of adding verified problems is priced in, the precision-optimal repeat count is $R^{\ast }=\sqrt{{\mu }_w{c}_p/({\sigma }_b^2{c}_g)}$, and the variance curve is flat around it: on this benchmark the decisive reason to repeat is not precision but identification---two to three runs suffice to estimate the components that the ceiling and the fixed-benchmark interval require. All three are measurements of one benchmark at one decoding temperature, and we state explicitly what they do and do not license.

arXiv comment17 pages, 4 figures, 11 tables. Substantially revised and retitled from v1; reframes the original repeated-run evaluation data as a variance decomposition, with expanded statistical validation and experimental details. Data and code: https://github.com/Yongxi-Zhou/leetcode-evaluator-data-output . Accepted at the NeurIPS 2026 Workshop on AI for Verifiable Coding (VeriCodeGen)