Comparative Analysis of Tokenization Algorithms for Low-Resource Language Dzongkha

Tandin Wangchuk, Tad Gonsalves

Large Language Models (LLMs) are gaining popularity and improving rapidly. Tokenizers are crucial components of natural language processing, especially for LLMs. Tokenizers break d…