DevBench: A Realistic, Developer-Informed Benchmark for Code Generation Models

Adarsh Kumarappan, Pareesa Ameneh Golnari, Wen Wen +5

DevBench is a telemetry-driven benchmark designed to evaluate Large Language Models (LLMs) on realistic code completion tasks. It includes 1,800 evaluation instances across six pro…