Exploring Software Naturalness through Neural Language Models

Abstract

The Software Naturalness hypothesis argues that programming languages can be understood through the same techniques used in natural language processing. We explore this hypothesis through the use of a pre-trained transformer-based language model to perform code analysis tasks. Present approaches to code analysis depend heavily on features derived from the Abstract Syntax Tree (AST) while our transformer-based language models work on raw source code. This work is the first to investigate whether such language models can discover AST features automatically. To achieve this, we introduce a sequence labeling task that directly probes the language models understanding of AST. Our results show that transformer based language models achieve high accuracy in the AST tagging task. Furthermore, we evaluate our model on a software vulnerability identification task. Importantly, we show that our approach obtains vulnerability identification results comparable to graph based approaches that rely heavily on compilers for feature extraction.

References in corpus (6)

• Distributed Representations of Words and Phrases and their Compositionality
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair
• CodeBERT: A Pre-Trained Model for Programming and Natural Languages
• Learning and Evaluating Contextual Embedding of Source Code
• Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks

Cited by in corpus (9)

• VulBERTa: Simplified Source Code Pre-Training for Vulnerability Detection
• Natural Attack for Pre-trained Models of Code
• GraphCodeBERT: Pre-training Code Representations with Data Flow
• Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Review
• On using distributed representations of source code for the detection of C security vulnerabilities
• D2A: A Dataset Built for AI-Based Vulnerability Detection Methods Using Differential Analysis
• GraphCode2Vec: Generic Code Embedding via Lexical and Program Dependence Analyses
• How could Neural Networks understand Programs?
• Assessing the Latent Automated Program Repair Capabilities of Large Language Models using Round-Trip Translation