VulScribeR: Exploring RAG-based Vulnerability Augmentation with LLMs

Abstract

Detecting vulnerabilities is vital for software security, yet deep learning-based vulnerability detectors (DLVD) face a data shortage, which limits their effectiveness. Data augmentation can potentially alleviate the data shortage, but augmenting vulnerable code is challenging and requires a generative solution that maintains vulnerability. Previous works have only focused on generating samples that contain single statements or specific types of vulnerabilities. Recently, large language models (LLMs) have been used to solve various code generation and comprehension tasks with inspiring results, especially when fused with retrieval augmented generation (RAG). Therefore, we propose VulScribeR, a novel LLM-based solution that leverages carefully curated prompt templates to augment vulnerable datasets. More specifically, we explore three strategies to augment both single and multi-statement vulnerabilities, with LLMs, namely Mutation, Injection, and Extension. Our extensive evaluation across four vulnerability datasets and DLVD models, using three LLMs, show that our approach beats two SOTA methods Vulgen and VGX, and Random Oversampling (ROS) by 27.48%, 27.93%, and 15.41% in f1-score with 5K generated vulnerable samples on average, and 53.84%, 54.10%, 69.90%, and 40.93% with 15K generated vulnerable samples. Our approach demonstrates its feasibility for large-scale data augmentation by generating 1K samples at as cheap as US$ 1.88.

Accepted by TOSEM; 26 pages, 6 figures, 8 tables, 3 prompt templates, 1 algorithm

References in corpus (9)

• A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly
• Diversity in Machine Learning
• Vulnerability Detection with Fine-grained Interpretations
• Comparing Code Explanations Created by Students and Large Language Models
• The impact of feature importance methods on the interpretation of defect classifiers
• Self-Supervised Contrastive Learning for Code Retrieval and Summarization via Semantic-Preserving Transformations
• LLMParser: An Exploratory Study on Using Large Language Models for Log Parsing
• What Makes Good In-context Demonstrations for Code Intelligence Tasks with LLMs?
• Impact of Discretization Noise of the Dependent variable on Machine Learning Classifiers in Software Engineering