Can We Generate Shellcodes via Natural Language? An Empirical Study

Abstract

Writing software exploits is an important practice for offensive security analysts to investigate and prevent attacks. In particular, shellcodes are especially time-consuming and a technical challenge, as they are written in assembly language. In this work, we address the task of automatically generating shellcodes, starting purely from descriptions in natural language, by proposing an approach based on Neural Machine Translation (NMT). We then present an empirical study using a novel dataset (Shellcode_IA32), which consists of 3,200 assembly code snippets of real Linux/x86 shellcodes from public databases, annotated using natural language. Moreover, we propose novel metrics to evaluate the accuracy of NMT at generating shellcodes. The empirical analysis shows that NMT can generate assembly code snippets from the natural language with high accuracy and that in many cases can generate entire shellcodes with no errors.

arXiv comment33 pages, 5 figures, 9 tables. To be published in Automated Software Engineering journal

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• Neural Machine Translation by Jointly Learning to Align and Translate
• Sequence to Sequence Learning with Neural Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• NLTK: The Natural Language Toolkit
• A Structured Self-attentive Sentence Embedding
• Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
• CodeSearchNet Challenge: Evaluating the State of Semantic Code Search
• Does BLEU Score Work for Code Migration?
• Learning to Generate Corrective Patches using Neural Machine Translation
• Meta-Evaluation of Translation Evaluation Methods: a systematic up-to-date overview
• Relevance Transformer: Generating Concise Code Snippets with Relevance Feedback
• Translation Quality Assessment: A Brief Survey on Manual and Automatic Methods

Cited by in corpus (5)

• Automating the Correctness Assessment of AI-generated Code for Security Contexts
• Who Evaluates the Evaluators? On Automatic Metrics for Assessing AI-based Offensive Code Generators
• AI Code Generators for Security: Friend or Foe?
• Enhancing Robustness of AI Offensive Code Generators via Data Augmentation
• Can NMT Understand Me? Towards Perturbation-based Evaluation of NMT Models for Code Generation