Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations

Evan Miller

Evaluations are critical for understanding the capabilities of large language models (LLMs). Fundamentally, evaluations are experiments; but the literature on evaluations has large…