Knowing When to Stop: Bayesian Optimal Stopping for LLM Evaluations

Toby D. Pilditch

LLM evaluations often use fixed sampling budgets, testing every item the same number of times even after estimates are precise. We introduce optstop, a precision-based adaptive sto…