CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
arXiv:2102.04664
Abstract
Benchmark datasets have a significant impact on accelerating research in programming language tasks. In this paper, we introduce CodeXGLUE, a benchmark dataset to foster machine learning research for program understanding and generation. CodeXGLUE includes a collection of 10 tasks across 14 datasets and a platform for model evaluation and comparison. CodeXGLUE also features three baseline systems, including the BERT-style, GPT-style, and Encoder-Decoder models, to make it easy for researchers to use the platform. The availability of such data and baselines can help the development and validation of new methods that can be applied to various program understanding and generation problems.
14 pages; Revise CodeBLEU scores for all models on text-to-code task
References in corpus (8)
- Sequence to Sequence Learning with Neural Networks
- CodeBLEU: a Method for Automatic Evaluation of Code Synthesis
- XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation
- Code Generation as a Dual Task of Code Summarization
- Unsupervised Translation of Programming Languages
- Detecting Code Clones with Graph Neural Networkand Flow-Augmented Abstract Syntax Tree
- Coupling Retrieval and Meta-Learning for Context-Dependent Semantic Parsing
- Semantic Scaffolds for Pseudocode-to-Code Generation