CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models

Abstract

Code generation models based on the pre-training and fine-tuning paradigm have been increasingly attempted by both academia and industry, resulting in well-known industrial models such as Codex, CodeGen, and PanGu-Coder. To evaluate the effectiveness of these models, multiple existing benchmarks are proposed, including only cases of generating a standalone function, i.e., a function that may invoke or access only built-in functions and standard libraries. However, non-standalone functions, which typically are not included in the existing benchmarks, constitute more than 70% of the functions in popular open-source projects, and evaluating models' effectiveness on standalone functions cannot reflect these models' effectiveness on pragmatic code generation scenarios. To help bridge the preceding gap, in this paper, we propose a benchmark named CoderEval, consisting of 230 Python and 230 Java code generation tasks carefully curated from popular real-world open-source projects and a self-contained execution platform to automatically assess the functional correctness of generated code. CoderEval supports code generation tasks from six levels of context dependency, where context refers to code elements such as types, APIs, variables, and consts defined outside the function under generation but within the dependent third-party libraries, current class, file, or project. CoderEval can be used to evaluate the effectiveness of models in generating code beyond only standalone functions. By evaluating three code generation models on CoderEval, we find that the effectiveness of these models in generating standalone functions is substantially higher than that in generating non-standalone functions. Our analysis highlights the current progress and pinpoints future directions to further improve a model's effectiveness by leveraging contextual information for pragmatic code generation.

References in corpus (19)

• Evaluating Large Language Models Trained on Code
• The Curious Case of Neural Text Degeneration
• A Deep Relevance Matching Model for Ad-hoc Retrieval
• Reinforcement Learning for Automatic Test Case Prioritization and Selection in Continuous Integration
• CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
• BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
• StarCoder: may the source be with you!
• Measuring Coding Challenge Competence With APPS
• InCoder: A Generative Model for Code Infilling and Synthesis
• DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence
• PanGu-$\alpha$: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation
• WizardCoder: Empowering Code Large Language Models with Evol-Instruct
• DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation
• Magicoder: Empowering Code Generation with OSS-Instruct
• PanGu-Coder: Program Synthesis with Function-Level Language Modeling
• What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?
• Training and Evaluating a Jupyter Notebook Data Science Assistant
• Language Models Can Teach Themselves to Program Better
• AixBench: A Code Generation Benchmark Dataset

Cited by in corpus (2)

• Automated Test Transfer Across Android Apps Using Large Language Models
• GUI Test Migration via Abstraction and Concretization