CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation

Abstract

Benchmark datasets have a significant impact on accelerating research in programming language tasks. In this paper, we introduce CodeXGLUE, a benchmark dataset to foster machine learning research for program understanding and generation. CodeXGLUE includes a collection of 10 tasks across 14 datasets and a platform for model evaluation and comparison. CodeXGLUE also features three baseline systems, including the BERT-style, GPT-style, and Encoder-Decoder models, to make it easy for researchers to use the platform. The availability of such data and baselines can help the development and validation of new methods that can be applied to various program understanding and generation problems.

14 pages; Revise CodeBLEU scores for all models on text-to-code task

References in corpus (24)

• Sequence to Sequence Learning with Neural Networks
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
• Release Strategies and the Social Impacts of Language Models
• Convolutional Neural Networks over Tree Structures for Programming Language Processing
• CodeBERT: A Pre-Trained Model for Programming and Natural Languages
• Neural Programmer-Interpreters
• CodeBLEU: a Method for Automatic Evaluation of Code Synthesis
• GraphCodeBERT: Pre-training Code Representations with Data Flow
• Structured Neural Summarization
• Mining Idioms from Source Code
• Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks
• A Syntactic Neural Model for General-Purpose Code Generation
• Neural Code Comprehension: A Learnable Representation of Code Semantics
• StaQC: A Systematically Mined Question-Code Dataset from Stack Overflow
• Neural Programmer: Inducing Latent Programs with Gradient Descent
• XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation
• Code Generation as a Dual Task of Code Summarization
• Unsupervised Translation of Programming Languages
• Detecting Code Clones with Graph Neural Networkand Flow-Augmented Abstract Syntax Tree
• TranS^3: A Transformer-based Framework for Unifying Code Summarization and Code Search
• Coupling Retrieval and Meta-Learning for Context-Dependent Semantic Parsing
• Unit Test Case Generation with Transformers and Focal Context
• Semantic Scaffolds for Pseudocode-to-Code Generation

Cited by in corpus (56)

• Evaluating Large Language Models Trained on Code
• Practical Program Repair in the Era of Large Pre-trained Language Models
• Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming
• WILDS: A Benchmark of in-the-Wild Distribution Shifts
• VulBERTa: Simplified Source Code Pre-Training for Vulnerability Detection
• Natural Attack for Pre-trained Models of Code
• No More Fine-Tuning? An Experimental Evaluation of Prompt Tuning in Code Intelligence
• Measuring Coding Challenge Competence With APPS
• Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Review
• Out of the BLEU: how should we assess quality of the Code Generation models?
• What Makes Good In-context Demonstrations for Code Intelligence Tasks with LLMs?
• Deep Learning-based Software Engineering: Progress, Challenges, and Opportunities
• LLM4PLC: Harnessing Large Language Models for Verifiable Programming of PLCs in Industrial Control Systems
• Code Search: A Survey of Techniques for Finding Code
• HPC-GPT: Integrating Large Language Model for High-Performance Computing
• Fully Autonomous Programming with Large Language Models
• Vul-LMGNNs: Fusing language models and online-distilled graph neural networks for code vulnerability detection
• Unified Pre-training for Program Understanding and Generation
• Diet Code Is Healthy: Simplifying Programs for Pre-trained Models of Code
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• On the Effectiveness of Transfer Learning for Code Search
• Revisiting the Plastic Surgery Hypothesis via Large Language Models
• A Comprehensive Review of State-of-The-Art Methods for Java Code Generation from Natural Language Text
• DOBF: A Deobfuscation Pre-Training Objective for Programming Languages
• The EarlyBIRD Catches the Bug: On Exploiting Early Layers of Encoder Models for More Efficient Code Classification
• On using distributed representations of source code for the detection of C security vulnerabilities
• DeepDebug: Fixing Python Bugs Using Stack Traces, Backtranslation, and Code Skeletons
• AVATAR: A Parallel Corpus for Java-Python Program Translation
• Toward a Theory of Causation for Interpreting Neural Code Models
• Assessing Evaluation Metrics for Neural Test Oracle Generation
• Diverse Title Generation for Stack Overflow Posts with Multiple Sampling Enhanced Transformer
• Matching Linear Algebra and Tensor Code to Specialized Hardware Accelerators
• The Good, the Bad, and the Missing: Neural Code Generation for Machine Learning Tasks
• Data Quality Issues in Vulnerability Detection Datasets
• Leveraging Automated Unit Tests for Unsupervised Code Translation
• Literature review on vulnerability detection using NLP technology
• Scaffolding Progress: How Structured Editors Shape Novice Errors When Transitioning from Blocks to Text
• Cascaded Fast and Slow Models for Efficient Semantic Code Search
• CoTexT: Multi-task Learning with Code-Text Transformer
• Enabling Automatic Repair of Source Code Vulnerabilities Using Data-Driven Methods
• ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generation
• Lyra: A Benchmark for Turducken-Style Code Generation
• Energy-Based Models for Code Generation under Compilability Constraints
• Variational Prefix Tuning for Diverse and Accurate Code Summarization Using Pre-trained Language Models
• PyTorrent: A Python Library Corpus for Large-scale Language Models
• An Empirical Study of the Imbalance Issue in Software Vulnerability Detection
• Poisoned Source Code Detection in Code Models
• Study of Distractors in Neural Models of Code
• A Comparison of Code Embeddings and Beyond
• Text2App: A Framework for Creating Android Apps from Text Descriptions
• CoSQA: 20,000+ Web Queries for Code Search and Question Answering
• SYNFIX: Automatically Fixing Syntax Errors using Compiler Diagnostics
• INSPECT: Intrinsic and Systematic Probing Evaluation for Code Transformers
• Programming Puzzles
• Neural Program Generation Modulo Static Analysis
• Code Representation Learning with Prüfer Sequences