Adapting Language Models via Token Translation

Zhili Feng, Tanya Marwah, Nicolo Fusi +2

Modern large language models use a fixed tokenizer to effectively compress text drawn from a source domain. However, applying the same tokenizer to a new target domain often leads…