How Different Tokenization Algorithms Impact LLMs and Transformer Models for Binary Code Analysis

Ahmed Mostafa, Raisul Arefin Nahid, Samuel Mulder

Tokenization is fundamental in assembly code analysis, impacting intrinsic characteristics like vocabulary size, semantic coverage, and extrinsic performance in downstream tasks. D…