LLM-as-a-Judge for Scalable Test Coverage Evaluation: Accuracy, Operational Reliability, and Cost

Donghao Huang; Shila Chew; Anna Dutkiewicz; Zhaoxia Wang

paper

LLM-as-a-Judge for Scalable Test Coverage Evaluation: Accuracy, Operational Reliability, and Cost

arXiv:2512.01232

Abstract

Assessing software test coverage at scale remains a bottleneck in QA pipelines. We present LLM-as-a-Judge (LAJ), a production-ready, rubric-driven framework for evaluating Gherkin acceptance tests with structured JSON outputs. Across 20 model configurations (GPT-4, GPT-5 with varying reasoning effort, and open-weight models) on 100 expert-annotated scripts over 5 runs (500 evaluations), we provide the first comprehensive analysis spanning accuracy, operational reliability, and cost. We introduce the Evaluation Completion Rate (ECR@1) to quantify first-attempt success, revealing reliability from 85.4% to 100.0% with material cost implications via retries. Results show that smaller models can outperform larger ones: GPT-4o Mini attains the best accuracy (6.07 MAAE), high reliability (96.6% ECR@1), and low cost ( $1.01 p er 1 K), y i e l d in g a 78 x cos t r e d u c t i o n v s . GP T - 5 (hi g h r e a so nin g) w hi l e im p r o v in g a cc u r a cy . R e a so nin g e f f or t i s m o d e l - f ami l y d e p e n d e n t : GP T - 5 b e n e f i t s f r o min cr e a se d r e a so nin g (w i t h p r e d i c t ab l e a cc u r a cy - cos tt r a d eo f f s), w h er e a so p e n - w e i g h t m o d e l s d e g r a d e a cr oss a l l d im e n s i o n s a sr e a so nin g in cr e a ses . O v er a l l, cos t s p an s 175 x ($ 0.45-$78.96 per 1K). We release the dataset, framework, and code to support reproducibility and deployment.

7 pages, accepted by the AAAI 2026 Workshop on Next Gen Code Development with Collaborative AI Agents