Modeling Vocabulary for Big Code Machine Learning
arXiv:1904.01873
Abstract
When building machine learning models that operate on source code, several decisions have to be made to model source-code vocabulary. These decisions can have a large impact: some can lead to not being able to train models at all, others significantly affect performance, particularly for Neural Language Models. Yet, these decisions are not often fully described. This paper lists important modeling choices for source code vocabulary, and explores their impact on the resulting vocabulary on a large-scale corpus of 14,436 projects. We show that a subset of decisions have decisive characteristics, allowing to train accurate Neural Language Models quickly on a large corpus of 10,106 projects.
12 pages, 1 figure
References in corpus (13)
- Adam: A Method for Stochastic Optimization
- Distributed Representations of Words and Phrases and their Compositionality
- Character-Aware Neural Language Models
- Exploring the Limits of Language Modeling
- Pointer Sentinel Mixture Models
- Regularizing and Optimizing LSTM Language Models
- Learning Natural Coding Conventions
- Quasi-Recurrent Neural Networks
- Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context
- Maybe Deep Neural Networks are the Best Choice for Modeling Source Code
- Context2Name: A Deep Learning-Based Approach to Infer Natural Variable Names from Usage Contexts
- Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow
- DeepAM: Migrate APIs with Multi-modal Sequence to Sequence Learning
Cited by in corpus (7)
- Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code
- On the Feasibility of Transfer-learning Code Smells using Deep Learning
- Neural Bug Finding: A Study of Opportunities and Challenges
- PYInfer: Deep Learning Semantic Type Inference for Python Variables
- On the Applicability of Language Models to Block-Based Programs
- Deep Generation of Coq Lemma Names Using Elaborated Terms
- Pointing to Subwords for Generating Function Names in Source Code