Unified Pre-training for Program Understanding and Generation

Abstract

Code summarization and generation empower conversion between programming language (PL) and natural language (NL), while code translation avails the migration of legacy code from one PL to another. This paper introduces PLBART, a sequence-to-sequence model capable of performing a broad spectrum of program and language understanding and generation tasks. PLBART is pre-trained on an extensive collection of Java and Python functions and associated NL text via denoising autoencoding. Experiments on code summarization in the English language, code generation, and code translation in seven programming languages show that PLBART outperforms or rivals state-of-the-art models. Moreover, experiments on discriminative tasks, e.g., program repair, clone detection, and vulnerable code detection, demonstrate PLBART's effectiveness in program understanding. Furthermore, analysis reveals that PLBART learns program syntax, style (e.g., identifier naming convention), logical flow (e.g., if block inside an else block is equivalent to else if block) that are crucial to program semantics and thus excels even with limited annotations.

NAACL 2021 (camera ready)

References in corpus (10)

• VisualBERT: A Simple and Performant Baseline for Vision and Language
• CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
• Release Strategies and the Social Impacts of Language Models
• CodeBLEU: a Method for Automatic Evaluation of Code Synthesis
• GraphCodeBERT: Pre-training Code Representations with Data Flow
• Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks
• CODIT: Code Editing with Tree-Based Neural Models
• Unsupervised Translation of Programming Languages
• Tree-Transformer: A Transformer-Based Method for Correction of Tree-Structured Data
• Program Language Translation Using a Grammar-Driven Tree-to-Tree Model

Cited by in corpus (5)

• Deep Learning-based Software Engineering: Progress, Challenges, and Opportunities
• Leveraging Automated Unit Tests for Unsupervised Code Translation
• Literature review on vulnerability detection using NLP technology
• ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generation
• How could Neural Networks understand Programs?