Self-Supervised Contrastive Learning for Code Retrieval and Summarization via Semantic-Preserving Transformations

Abstract

We propose Corder, a self-supervised contrastive learning framework for source code model. Corder is designed to alleviate the need of labeled data for code retrieval and code summarization tasks. The pre-trained model of Corder can be used in two ways: (1) it can produce vector representation of code which can be applied to code retrieval tasks that do not have labeled data; (2) it can be used in a fine-tuning process for tasks that might still require label data such as code summarization. The key innovation is that we train the source code model by asking it to recognize similar and dissimilar code snippets through a contrastive learning objective. To do so, we use a set of semantic-preserving transformation operators to generate code snippets that are syntactically diverse but semantically equivalent. Through extensive experiments, we have shown that the code models pretrained by Corder substantially outperform the other baselines for code-to-code retrieval, text-to-code retrieval, and code-to-text summarization tasks.

Accepted at SIGIR 2021

Cited by in corpus (14)

• Deep Learning-based Software Engineering: Progress, Challenges, and Opportunities
• SelfAPR: Self-supervised Program Repair with Test Execution Diagnostics
• Code Search based on Context-aware Code Translation
• CodeMark: Imperceptible Watermarking for Code Datasets against Neural Code Completion Models
• Towards Causal Deep Learning for Vulnerability Detection
• Survey of Code Search Based on Deep Learning
• VulScribeR: Exploring RAG-based Vulnerability Augmentation with LLMs
• FuzzSlice: Pruning False Positives in Static Analysis Warnings Through Function-Level Fuzzing
• The Good, the Bad, and the Missing: Neural Code Generation for Machine Learning Tasks
• Self-Supervised Learning to Prove Equivalence Between Straight-Line Programs via Rewrite Rules
• NeuDep: Neural Binary Memory Dependence Analysis
• A Closer Look into Transformer-Based Code Intelligence Through Code Transformation: Challenges and Opportunities
• Deep Code Search with Naming-Agnostic Contrastive Multi-View Learning
• UNICS: Multilingual Code Search via Unified Pseudocode and Contrastive Transfer Learning