Structural Language Models of Code

Abstract

We address the problem of any-code completion - generating a missing piece of source code in a given program without any restriction on the vocabulary or structure. We introduce a new approach to any-code completion that leverages the strict syntax of programming languages to model a code snippet as a tree - structural language modeling (SLM). SLM estimates the probability of the program's abstract syntax tree (AST) by decomposing it into a product of conditional probabilities over its nodes. We present a neural model that computes these conditional probabilities by considering all AST paths leading to a target node. Unlike previous techniques that have severely restricted the kinds of expressions that can be generated in this task, our approach can generate arbitrary code in any programming language. Our model significantly outperforms both seq2seq and a variety of structured approaches in generating Java and C# code. Our code, data, and trained models are available at http://github.com/tech-srl/slm-code-generation/ . An online demo is available at http://AnyCodeGen.org .

arXiv commentAppeared in ICML'2020

Cited by in corpus (13)

• GraphCodeBERT: Pre-training Code Representations with Data Flow
• Pretrained Transformers for Text Ranking: BERT and Beyond
• An Empirical Study on the Usage of Transformer Models for Code Completion
• Deep Graph Matching and Searching for Semantic Code Retrieval
• You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion
• Code Prediction by Feeding Trees to Transformers
• Beep: Fine-grained Fix Localization by Learning to Predict Buggy Code Elements
• Learning Autocompletion from Real-World Datasets
• A Comparison of Code Embeddings and Beyond
• How could Neural Networks understand Programs?
• Learning to Extend Program Graphs to Work-in-Progress Code
• Capturing Structural Locality in Non-parametric Language Models
• Programming Puzzles