Language Modelling for Source Code with Transformer-XL
arXiv:2007.15813
Abstract
It has been found that software, like natural language texts, exhibits "naturalness", which can be captured by statistical language models. In recent years, neural language models have been proposed to represent the naturalness of software through deep learning. In this paper, we conduct an experimental evaluation of state-of-the-art neural language models for source code, including RNN-based models and Transformer-XL based models. Through experiments on a large-scale Python code corpus, we find that the Transformer-XL model outperforms RNN-based models (including LSTM and GRU models) in capturing the naturalness of software, with far less computational cost.
References in corpus (8)
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- Gaussian Error Linear Units (GELUs)
- Learning Natural Coding Conventions
- Proposal of room-temperature diamond maser
- A deep language model for software code
- Automated Correction for Syntax Errors in Programming Assignments using Recurrent Neural Networks
- Learning Python Code Suggestion with a Sparse Pointer Network
- Maybe Deep Neural Networks are the Best Choice for Modeling Source Code