Getting pwn'd by AI: Penetration Testing with Large Language Models

Abstract

The field of software security testing, more specifically penetration testing, is an activity that requires high levels of expertise and involves many manual testing and analysis steps. This paper explores the potential usage of large-language models, such as GPT3.5, to augment penetration testers with AI sparring partners. We explore the feasibility of supplementing penetration testers with AI models for two distinct use cases: high-level task planning for security testing assignments and low-level vulnerability hunting within a vulnerable virtual machine. For the latter, we implemented a closed-feedback loop between LLM-generated low-level actions with a vulnerable virtual machine (connected through SSH) and allowed the LLM to analyze the machine state for vulnerabilities and suggest concrete attack vectors which were automatically executed within the virtual machine. We discuss promising initial results, detail avenues for improvement, and close deliberating on the ethics of providing AI-based sparring partners.

References in corpus (2)

• Learning to Prompt for Vision-Language Models
• Understanding Hackers' Work: An Empirical Study of Offensive Security Practitioners

Cited by in corpus (13)

• A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly
• LLM for SoC Security: A Paradigm Shift
• Generative AI for Self-Adaptive Systems: State of the Art and Research Roadmap
• Just another copy and paste? Comparing the security vulnerabilities of ChatGPT generated code and StackOverflow answers
• Understanding Hackers' Work: An Empirical Study of Offensive Security Practitioners
• CIPHER: Cybersecurity Intelligent Penetration-testing Helper for Ethical Researcher
• LLMs as Hackers: Autonomous Linux Privilege Escalation Attacks
• Out of the Cage: How Stochastic Parrots Win in Cyber Security Environments
• Can LLMs Hack Enterprise Networks? Autonomous Assumed Breach Penetration-Testing Active Directory Networks
• Towards Supporting Penetration Testing Education with Large Language Models: an Evaluation and Comparison
• Survey on AI Ethics: A Socio-technical Perspective
• Supporting Artifact Evaluation with LLMs: A Study with Published Security Research Papers
• Weaponizing Language Models for Cybersecurity Offensive Operations: Automating Vulnerability Assessment Report Validation; A Review Paper