Maybe Deep Neural Networks are the Best Choice for Modeling Source Code

Abstract

Statistical language modeling techniques have successfully been applied to source code, yielding a variety of new software development tools, such as tools for code suggestion and improving readability. A major issue with these techniques is that code introduces new vocabulary at a far higher rate than natural language, as new identifier names proliferate. But traditional language models limit the vocabulary to a fixed set of common words. For code, this strong assumption has been shown to have a significant negative effect on predictive performance. But the open vocabulary version of the neural network language models for code have not been introduced in the literature. We present a new open-vocabulary neural language model for code that is not limited to a fixed vocabulary of identifier names. We employ a segmentation into subword units, subsequences of tokens chosen based on a compression criterion, following previous work in machine translation. Our network achieves best in class performance, outperforming even the state-of-the-art methods of Hellendoorn and Devanbu that are designed specifically to model code. Furthermore, we present a simple method for dynamically adapting the model to a new test project, resulting in increased performance. We showcase our methodology on code corpora in three different languages of over a billion tokens each, hundreds of times larger than in previous work. To our knowledge, this is the largest neural language model for code that has been reported.

References in corpus (12)

• Distributed Representations of Words and Phrases and their Compositionality
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• An Empirical Study of Smoothing Techniques for Language Modeling
• Learning Natural Coding Conventions
• Hierarchical Multiscale Recurrent Neural Networks
• Efficient softmax approximation for GPUs
• A Convolutional Attention Network for Extreme Summarization of Source Code
• Automated Correction for Syntax Errors in Programming Assignments using Recurrent Neural Networks
• A deep language model for software code
• Context2Name: A Deep Learning-Based Approach to Infer Natural Variable Names from Usage Contexts
• Learning to Create and Reuse Words in Open-Vocabulary Neural Language Modeling
• Program Synthesis using Natural Language

Cited by in corpus (18)

• SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair
• Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code
• On the Feasibility of Transfer-learning Code Smells using Deep Learning
• An Empirical Study on the Usage of Transformer Models for Code Completion
• On the Generalizability of Neural Program Models with respect to Semantic-Preserving Program Transformations
• On the Effectiveness of Transfer Learning for Code Search
• Modeling Vocabulary for Big Code Machine Learning
• Sequence Model Design for Code Completion in the Modern IDE
• Toward a Theory of Causation for Interpreting Neural Code Models
• PYInfer: Deep Learning Semantic Type Inference for Python Variables
• Evaluation of Generalizability of Neural Program Analyzers under Semantic-Preserving Transformations
• Language Modelling for Source Code with Transformer-XL
• Learning Autocompletion from Real-World Datasets
• Automated Source Code Generation and Auto-completion Using Deep Learning: Comparing and Discussing Current Language-Model-Related Approaches
• Using Sequence-to-Sequence Learning for Repairing C Vulnerabilities
• On-the-Fly Adaptation of Source Code Models using Meta-Learning
• Modeling Programs Hierarchically with Stack-Augmented LSTM
• Learning to Extend Program Graphs to Work-in-Progress Code