PalmTree: Learning an Assembly Language Model for Instruction Embedding

Abstract

Deep learning has demonstrated its strengths in numerous binary analysis tasks, including function boundary detection, binary code search, function prototype inference, value set analysis, etc. When applying deep learning to binary analysis tasks, we need to decide what input should be fed into the neural network model. More specifically, we need to answer how to represent an instruction in a fixed-length vector. The idea of automatically learning instruction representations is intriguing, however the existing schemes fail to capture the unique characteristics of disassembly. These schemes ignore the complex intra-instruction structures and mainly rely on control flow in which the contextual information is noisy and can be influenced by compiler optimizations. In this paper, we propose to pre-train an assembly language model called PalmTree for generating general-purpose instruction embeddings by conducting self-supervised training on large-scale unlabeled binary corpora. PalmTree utilizes three pre-training tasks to capture various characteristics of assembly language. These training tasks overcome the problems in existing schemes, thus can help to generate high-quality representations. We conduct both intrinsic and extrinsic evaluations, and compare PalmTree with other instruction embedding schemes. PalmTree has the best performance for intrinsic metrics, and outperforms the other instruction embedding schemes for all downstream tasks.

References in corpus (5)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Pre-trained Models for Natural Language Processing: A Survey
• Neural Network-based Graph Embedding for Cross-Platform Binary Code Similarity Detection
• Neural Machine Translation Inspired Binary Code Similarity Comparison beyond Function Pairs
• Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code

Cited by in corpus (11)

• Beyond the C: Retargetable Decompilation using Neural Machine Translation
• Bi-Directional Transformers vs. word2vec: Discovering Vulnerabilities in Lifted Compiled Code
• NeuDep: Neural Binary Memory Dependence Analysis
• Black-box Attacks Against Neural Binary Function Detection
• Enhancing Function Name Prediction using Votes-Based Name Tokenization and Multi-Task Learning
• ORCAS: Obfuscation-Resilient Binary Code Similarity Analysis using Dominance Enhanced Semantic Graph
• Multi-modal Learning for WebAssembly Reverse Engineering
• Analyzing PDFs like Binaries: Adversarially Robust PDF Malware Analysis via Intermediate Representation and Language Model
• Deep Learning-based Binary Analysis for Vulnerability Detection in x86-64 Machine Code
• Pretraining on Call Graphs: When Binary Analysis Tasks Profit From Context
• REx86: A Local Large Language Model for Assisting in x86 Assembly Reverse Engineering