Hybrid Tokenization Strategy for DNA Language Model using Byte Pair Encoding and K-MER Methods

Ganesh Sapkota, Md Hasibur Rahman

This paper presents a novel hybrid tokenization strategy that enhances the performance of DNA Language Models (DLMs) by combining 6-mer tokenization with Byte Pair Encoding (BPE-60…