collaborators

1 paper

cs.CL2026

Writing-System-Level Tokenizer Adaptation for Byte-Level BPE

Bohdan Didenko

Pretrained byte-level BPE tokenizers can segment underrepresented languages inefficiently. Replacing a tokenizer changes the meaning of nearly every token ID, while vocabulary expa…