How Much Can Repeated Running Buy? A Measured Variance Ceiling for LLM Code Evaluation

Yongxi Zhou, Lai Yun Choi, Jiaxi Wen +2

A benchmark score is a measurement whose uncertainty has two sources: an aleatoric component from stochastic decoding and an epistemic component from the finite problem set. Using…