Beyond Atomic Tokens: Factorizing Syllables for Language Model Pretraining

Nghia Hieu Nguyen, Thai Bao Huynh, Binh-An Dinh-Le +4

Conventional tokenizers represent text as characters or statistically derived subwords, overlooking the internal phonological structure of syllables and often requiring large vocab…