DOBF: A Deobfuscation Pre-Training Objective for Programming Languages
arXiv:2102.07492
Abstract
Recent advances in self-supervised learning have dramatically improved the state of the art on a wide variety of tasks. However, research in language model pre-training has mostly focused on natural languages, and it is unclear whether models like BERT and its variants provide the best pre-training when applied to other modalities, such as source code. In this paper, we introduce a new pre-training objective, DOBF, that leverages the structural aspect of programming languages and pre-trains a model to recover the original version of obfuscated source code. We show that models pre-trained with DOBF significantly outperform existing approaches on multiple downstream tasks, providing relative improvements of up to 13% in unsupervised code translation, and 24% in natural language code search. Incidentally, we found that our pre-trained model is able to de-obfuscate fully obfuscated source files, and to suggest descriptive variable names.
References in corpus (17)
- Adam: A Method for Stochastic Optimization
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Cross-lingual Language Model Pretraining
- Unified Language Model Pre-training for Natural Language Understanding and Generation
- ERNIE: Enhanced Representation through Knowledge Integration
- MASS: Masked Sequence to Sequence Pre-training for Language Generation
- CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
- SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair
- Learning Natural Coding Conventions
- CodeBERT: A Pre-Trained Model for Programming and Natural Languages
- Unsupervised Machine Translation Using Monolingual Corpora Only
- Learning and Evaluating Contextual Embedding of Source Code
- Contrastive Code Representation Learning
- Neural Reverse Engineering of Stripped Binaries using Augmented Control Flow Graphs
- Dynamic Neural Program Embedding for Program Repair
- Context2Name: A Deep Learning-Based Approach to Infer Natural Variable Names from Usage Contexts
- Code Prediction by Feeding Trees to Transformers
Cited by in corpus (4)
- CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation
- Unsupervised Neural Machine Translation with Generative Language Models Only
- Leveraging Automated Unit Tests for Unsupervised Code Translation
- SYNFIX: Automatically Fixing Syntax Errors using Compiler Diagnostics