CodeBLEU: a Method for Automatic Evaluation of Code Synthesis

Abstract

Evaluation metrics play a vital role in the growth of an area as it defines the standard of distinguishing between good and bad models. In the area of code synthesis, the commonly used evaluation metric is BLEU or perfect accuracy, but they are not suitable enough to evaluate codes, because BLEU is originally designed to evaluate the natural language, neglecting important syntactic and semantic features of codes, and perfect accuracy is too strict thus it underestimates different outputs with the same semantic logic. To remedy this, we introduce a new automatic evaluation metric, dubbed CodeBLEU. It absorbs the strength of BLEU in the n-gram match and further injects code syntax via abstract syntax trees (AST) and code semantics via data-flow. We conduct experiments by evaluating the correlation coefficient between CodeBLEU and quality scores assigned by the programmers on three code synthesis tasks, i.e., text-to-code, code translation, and code refinement. Experimental results show that our proposed CodeBLEU can achieve a better correlation with programmer assigned scores compared with BLEU and accuracy.

8 pages, 6 figures

References in corpus (8)

• Neural Machine Translation by Jointly Learning to Align and Translate
• Sequence to Sequence Learning with Neural Networks
• CodeBERT: A Pre-Trained Model for Programming and Natural Languages
• GraphCodeBERT: Pre-training Code Representations with Data Flow
• Learning and Evaluating Contextual Embedding of Source Code
• Unsupervised Translation of Programming Languages
• A parallel corpus of Python functions and documentation strings for automated code documentation and code generation
• Coupling Retrieval and Meta-Learning for Context-Dependent Semantic Parsing

Cited by in corpus (25)

• Evaluating Large Language Models Trained on Code
• CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
• No More Fine-Tuning? An Experimental Evaluation of Prompt Tuning in Code Intelligence
• Measuring Coding Challenge Competence With APPS
• Out of the BLEU: how should we assess quality of the Code Generation models?
• An Empirical Study on the Usage of Transformer Models for Code Completion
• What Makes Good In-context Demonstrations for Code Intelligence Tasks with LLMs?
• Fully Autonomous Programming with Large Language Models
• Unified Pre-training for Program Understanding and Generation
• Automating the Correctness Assessment of AI-generated Code for Security Contexts
• CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation
• A Comprehensive Review of State-of-The-Art Methods for Java Code Generation from Natural Language Text
• AVATAR: A Parallel Corpus for Java-Python Program Translation
• Assessing Evaluation Metrics for Neural Test Oracle Generation
• Metamorphic Malware Evolution: The Potential and Peril of Large Language Models
• The Good, the Bad, and the Missing: Neural Code Generation for Machine Learning Tasks
• VISION: A Modular AI Assistant for Natural Human-Instrument Interaction at Scientific User Facilities
• Unified Modeling Language Code Generation from Diagram Images Using Multimodal Large Language Models
• Program Repair with Minimal Edits Using CodeT5
• CoTexT: Multi-task Learning with Code-Text Transformer
• Fully Autonomous Programming using Iterative Multi-Agent Debugging with Large Language Models
• Enabling Automatic Repair of Source Code Vulnerabilities Using Data-Driven Methods
• Programming Puzzles
• Towards A Measure Of General Machine Intelligence
• Jointly Learning to Repair Code and Generate Commit Message