#token fragmentation

topictoken fragmentation

1 paper · 1 filter

cs.CL2026

Language Models are not Equally Robust to Non-Canonical Tokenization across Languages

Poulami Ghosh, Preethi Jyothi

The paper investigates how language models handle non-canonical tokenizations across 27 languages, finding that robustness varies by language and token fragmentation, and shows tha…