TreeGen: A Tree-Based Transformer Architecture for Code Generation

Abstract

A code generation system generates programming language code based on an input natural language description. State-of-the-art approaches rely on neural networks for code generation. However, these code generators suffer from two problems. One is the long dependency problem, where a code element often depends on another far-away code element. A variable reference, for example, depends on its definition, which may appear quite a few lines before. The other problem is structure modeling, as programs contain rich structural information. In this paper, we propose a novel tree-based neural architecture, TreeGen, for code generation. TreeGen uses the attention mechanism of Transformers to alleviate the long-dependency problem, and introduces a novel AST reader (encoder) to incorporate grammar rules and AST structures into the network. We evaluated TreeGen on a Python benchmark, HearthStone, and two semantic parsing benchmarks, ATIS and GEO. TreeGen outperformed the previous state-of-the-art approach by 4.5 percentage points on HearthStone, and achieved the best accuracy among neural network-based approaches on ATIS (89.1%) and GEO (89.6%). We also conducted an ablation test to better understand each component of our model.

References in corpus (11)

• Learning to Map Sentences to Logical Form: Structured Classification with Probabilistic Categorial Grammars
• Convolutional Neural Networks over Tree Structures for Programming Language Processing
• Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
• Language to Logical Form with Neural Attention
• A Syntactic Neural Model for General-Purpose Code Generation
• Latent Predictor Networks for Code Generation
• Coarse-to-Fine Decoding for Neural Semantic Parsing
• Sequence-to-Action: End-to-End Semantic Graph Generation for Semantic Parsing
• Exploiting Rich Syntactic Information for Semantic Parsing with Graph-to-Sequence Model
• TRANX: A Transition-based Neural Abstract Syntax Parser for Semantic Parsing and Code Generation
• Retrieval-Based Neural Code Generation

Cited by in corpus (4)

• Deep Learning-based Software Engineering: Progress, Challenges, and Opportunities
• Embedding API Dependency Graph for Neural Code Generation
• Automatic Code Generation using Pre-Trained Language Models
• CoEdPilot: Recommending Code Edits with Learned Prior Edit Relevance, Project-wise Awareness, and Interactive Nature