Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the data

Florian E. Dorner, Vivian Y. Nastl, Moritz Hardt

High quality annotations are increasingly a bottleneck in the explosively growing machine learning ecosystem. Scalable evaluation methods that avoid costly annotation have therefor…