Measuring Five-Nines Reliability: Sample-Efficient LLM Evaluation in Saturated Benchmarks

Eungyeup Kim, Chenchen Gu, Vashisth Tiwari +1

While existing benchmarks demonstrate the near-perfect performance of large language models (LLMs) on various tasks, this apparent saturation often obscures the need for rigorous e…