SCELMo: Source Code Embeddings from Language Models
arXiv:2004.13214
Abstract
Continuous embeddings of tokens in computer programs have been used to support a variety of software development tools, including readability, code search, and program repair. Contextual embeddings are common in natural language processing but have not been previously applied in software engineering. We introduce a new set of deep contextualized word representations for computer programs based on language models. We train a set of embeddings using the ELMo (embeddings from language models) framework of Peters et al (2018). We investigate whether these embeddings are effective when fine-tuned for the downstream task of bug detection. We show that even a low-dimensional embedding trained on a relatively small corpus of programs can improve a state-of-the-art machine learning system for bug detection.
12 pages
References in corpus (7)
- Distributed Representations of Words and Phrases and their Compositionality
- SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair
- Sorting and Transforming Program Repair Ingredients via Deep Learning Code Similarities
- Automated software vulnerability detection with machine learning
- A Literature Study of Embeddings on Source Code
- Neural Program Repair by Jointly Learning to Localize and Repair
- Context2Name: A Deep Learning-Based Approach to Infer Natural Variable Names from Usage Contexts