LLM vs. SAST: A Technical Analysis on Detecting Coding Bugs of GPT4-Advanced Data Analysis
arXiv:2506.15212
The paper conducts a controlled study comparing GPT-4 (Advanced Data Analysis) with two SAST tools on 32 security scenarios, finding that GPT-4 detects bugs in 93.75% of cases versus 34% for the SAST baseline, with statistically significant results.
Abstract
Large language models (LLMs) are increasingly used for code understanding, yet their practical effectiveness for vulnerability detection relative to Static Application Security Testing (SAST) remains insufficiently quantified. We present a controlled comparative study between GPT-4 (Advanced Data Analysis) and two SAST tools (SonarQube and Cloud Defence) on 32 curated security scenarios representing common coding pitfalls. Each scenario is scored with a binary detection rule, the two SAST outputs are aggregated using a logical OR baseline, and paired outcomes are evaluated using McNemar's test for statistical significance. In our dataset, GPT-4 correctly detected 30 of 32 scenarios (93.75\%), while the aggregated SAST baseline detected 11 of 32. The paired comparison shows a statistically significant difference in detection performance in favour of GPT-4. We also discuss security considerations and operational constraints for integrating LLM-enhanced vulnerability scanning into secure software development workflows.