How Effective Are Neural Networks for Fixing Security Vulnerabilities

Abstract

Security vulnerability repair is a difficult task that is in dire need of automation. Two groups of techniques have shown promise: (1) large code language models (LLMs) that have been pre-trained on source code for tasks such as code completion, and (2) automated program repair (APR) techniques that use deep learning (DL) models to automatically fix software bugs. This paper is the first to study and compare Java vulnerability repair capabilities of LLMs and DL-based APR models. The contributions include that we (1) apply and evaluate five LLMs (Codex, CodeGen, CodeT5, PLBART and InCoder), four fine-tuned LLMs, and four DL-based APR techniques on two real-world Java vulnerability benchmarks (Vul4J and VJBench), (2) design code transformations to address the training and test data overlapping threat to Codex, (3) create a new Java vulnerability repair benchmark VJBench, and its transformed version VJBench-trans and (4) evaluate LLMs and APR techniques on the transformed vulnerabilities in VJBench-trans. Our findings include that (1) existing LLMs and APR models fix very few Java vulnerabilities. Codex fixes 10.2 (20.4%), the most number of vulnerabilities. (2) Fine-tuning with general APR data improves LLMs' vulnerability-fixing capabilities. (3) Our new VJBench reveals that LLMs and APR models fail to fix many Common Weakness Enumeration (CWE) types, such as CWE-325 Missing cryptographic step and CWE-444 HTTP request smuggling. (4) Codex still fixes 8.3 transformed vulnerabilities, outperforming all the other LLMs and APR models on transformed vulnerabilities. The results call for innovations to enhance automated Java vulnerability repair such as creating larger vulnerability repair training data, tuning LLMs with such data, and applying code simplification transformation to facilitate vulnerability repair.

This paper was accepted in the proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2023), and was presented at the conference, that was held in Seattle, USA, 17-21 July 2023

References in corpus (8)

• Evaluating Large Language Models Trained on Code
• CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
• InCoder: A Generative Model for Code Infilling and Synthesis
• Bears: An Extensible Java Bug Benchmark for Automatic Program Repair Studies
• AI-driven Development Is Here: Should You Worry?
• The Race to the Vulnerable: Measuring the Log4j Shell Incident
• DeepDebug: Fixing Python Bugs Using Stack Traces, Backtranslation, and Code Skeletons
• Automated Repair of Programs from Large Language Models

Cited by in corpus (7)

• LLM for SoC Security: A Paradigm Shift
• Harnessing the Power of LLM to Support Binary Taint Analysis
• LPR: Large Language Models-Aided Program Reduction
• Validating Network Protocol Parsers with Traceable RFC Document Interpretation
• Code Vulnerability Repair with Large Language Model using Context-Aware Prompt Tuning
• SPVR: syntax-to-prompt vulnerability repair based on large language models
• Seeing is Fixing: Cross-Modal Reasoning with Multimodal LLMs for Visual Software Issue Fixing