Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation

Philipp D. Siedler, Jordan Sassoon

Benchmark datasets are central to evaluating Large Language Models (LLMs), yet they are typically conceived as monolithic tasks, obscuring substantial variation in the demands of i…