Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code

Abstract

Statistical language modeling techniques have successfully been applied to large source code corpora, yielding a variety of new software development tools, such as tools for code suggestion, improving readability, and API migration. A major issue with these techniques is that code introduces new vocabulary at a far higher rate than natural language, as new identifier names proliferate. Both large vocabularies and out-of-vocabulary issues severely affect Neural Language Models (NLMs) of source code, degrading their performance and rendering them unable to scale. In this paper, we address this issue by: 1) studying how various modelling choices impact the resulting vocabulary on a large-scale corpus of 13,362 projects; 2) presenting an open vocabulary source code NLM that can scale to such a corpus, 100 times larger than in previous work; and 3) showing that such models outperform the state of the art on three distinct code corpora (Java, C, Python). To our knowledge, these are the largest NLMs for code that have been reported. All datasets, code, and trained models used in this work are publicly available.

13 pages; to appear in Proceedings of ICSE 2020

References in corpus (15)

• Distributed Representations of Words and Phrases and their Compositionality
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Character-Aware Neural Language Models
• Exploring the Limits of Language Modeling
• Pointer Sentinel Mixture Models
• SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair
• Learning Natural Coding Conventions
• SWIM: Synthesizing What I Mean
• Code Completion with Neural Attention and Pointer Networks
• Improving Neural Language Models with a Continuous Cache
• A deep language model for software code
• Automated Correction for Syntax Errors in Programming Assignments using Recurrent Neural Networks
• Maybe Deep Neural Networks are the Best Choice for Modeling Source Code
• Modeling Vocabulary for Big Code Machine Learning
• Splitting source code identifiers using Bidirectional LSTM Recurrent Neural Network

Cited by in corpus (26)

• Natural Attack for Pre-trained Models of Code
• PalmTree: Learning an Assembly Language Model for Instruction Embedding
• Neural Program Repair with Execution-based Backpropagation
• An Empirical Study on the Usage of Transformer Models for Code Completion
• On the Evaluation of Neural Code Summarization
• CodeFill: Multi-token Code Completion by Jointly Learning from Structure and Naming Sequences
• Recommending Metamodel Concepts during Modeling Activities with Pre-Trained Language Models
• Empirical Study of Transformers for Source Code
• Program Merge Conflict Resolution via Neural Transformers
• Greening Large Language Models of Code
• How Often Do Single-Statement Bugs Occur? The ManySStuBs4J Dataset
• Passport: Improving Automated Formal Verification Using Identifiers
• Predictive Mutation Analysis via Natural Language Channel in Source Code
• Learning Lenient Parsing & Typing via Indirect Supervision
• PYInfer: Deep Learning Semantic Type Inference for Python Variables
• T5APR: Empowering Automated Program Repair across Languages through Checkpoint Ensemble
• Towards Full-line Code Completion with Neural Language Models
• AsserT5: Test Assertion Generation Using a Fine-Tuned Code Language Model
• On the Applicability of Language Models to Block-Based Programs
• Advanced Graph-Based Deep Learning for Probabilistic Type Inference
• Patching as Translation: the Data and the Metaphor
• Better Modeling the Programming World with Code Concept Graphs-augmented Multi-modal Learning
• NaturalCC: A Toolkit to Naturalize the Source Code Corpus
• Cross-Domain Evaluation of a Deep Learning-Based Type Inference System
• Learning to Extend Program Graphs to Work-in-Progress Code
• Evaluation of Cross-Lingual Bug Localization: Two Industrial Cases