Rethinking Uncertainty Evaluation in Large Language Models

Krish Matta, Atharv Naphade, Andy Zou

Calibration is the primary criterion for evaluating LLM confidence, but it is insufficient: it admits trivially incoherent estimators, depends on the evaluation distribution, and d…