SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representation

Abstract

Code representation learning, which aims to encode the semantics of source code into distributed vectors, plays an important role in recent deep-learning-based models for code intelligence. Recently, many pre-trained language models for source code (e.g., CuBERT and CodeBERT) have been proposed to model the context of code and serve as a basis for downstream code intelligence tasks such as code search, code clone detection, and program translation. Current approaches typically consider the source code as a plain sequence of tokens, or inject the structure information (e.g., AST and data-flow) into the sequential model pre-training. To further explore the properties of programming languages, this paper proposes SynCoBERT, a syntax-guided multi-modal contrastive pre-training approach for better code representations. Specially, we design two novel pre-training objectives originating from the symbolic and syntactic properties of source code, i.e., Identifier Prediction (IP) and AST Edge Prediction (TEP), which are designed to predict identifiers, and edges between two nodes of AST, respectively. Meanwhile, to exploit the complementary information in semantically equivalent modalities (i.e., code, comment, AST) of the code, we propose a multi-modal contrastive learning strategy to maximize the mutual information among different modalities. Extensive experiments on four downstream tasks related to code intelligence show that SynCoBERT advances the state-of-the-art with the same pre-training corpus and model size.

arXiv comment9 pages, 3 figures, 5 tables

References in corpus (10)

• Convolutional Neural Networks over Tree Structures for Programming Language Processing
• CLEAR: Contrastive Learning for Sentence Representation
• Learning and Evaluating Contextual Embedding of Source Code
• Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks
• Contrastive Code Representation Learning
• DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• Language-Agnostic Representation Learning of Source Code from Structure and Context
• ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
• On using distributed representations of source code for the detection of C security vulnerabilities
• CoTexT: Multi-task Learning with Code-Text Transformer

Cited by in corpus (5)

• Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Review
• Survey of Code Search Based on Deep Learning
• An Unbiased Transformer Source Code Learning with Semantic Vulnerability Graph
• Cascaded Fast and Slow Models for Efficient Semantic Code Search
• UNICS: Multilingual Code Search via Unified Pseudocode and Contrastive Transfer Learning