Evaluation of Generalizability of Neural Program Analyzers under Semantic-Preserving Transformations

Abstract

The abundance of publicly available source code repositories, in conjunction with the advances in neural networks, has enabled data-driven approaches to program analysis. These approaches, called neural program analyzers, use neural networks to extract patterns in the programs for tasks ranging from development productivity to program reasoning. Despite the growing popularity of neural program analyzers, the extent to which their results are generalizable is unknown. In this paper, we perform a large-scale evaluation of the generalizability of two popular neural program analyzers using seven semantically-equivalent transformations of programs. Our results caution that in many cases the neural program analyzers fail to generalize well, sometimes to programs with negligible textual differences. The results provide the initial stepping stones for quantifying robustness in neural program analyzers.

Related to arXiv:2008.01566

References in corpus (7)

• Understanding Neural Networks through Representation Erasure
• Adversarial Examples for Evaluating Reading Comprehension Systems
• Maybe Deep Neural Networks are the Best Choice for Modeling Source Code
• COSET: A Benchmark for Evaluating Neural Program Embeddings
• Learning Blended, Precise Semantic Program Embeddings
• Learning Scalable and Precise Representation of Program Semantics
• Testing Neural Program Analyzers

Cited by in corpus (5)

• Contrastive Code Representation Learning
• On the Generalizability of Neural Program Models with respect to Semantic-Preserving Program Transformations
• Understanding Neural Code Intelligence Through Program Simplification
• TreeCaps: Tree-Based Capsule Networks for Source Code Processing
• Generating Adversarial Computer Programs using Optimized Obfuscations