Isolating Language-Coding from Problem-Solving: Benchmarking LLMs with PseudoEval

Jiarong Wu, Songqiang Chen, Jialun Cao +2

Existing code generation benchmarks for Large Language Models (LLMs) such as HumanEval and MBPP are designed to study LLMs' end-to-end performance, where the benchmarks feed a prob…