works on
embedding initialization 1model adaptation 1multilingual tokenization 1pretrained language models 1tokenizer expansion 1

From the 1 of 3 linked papers with an AI index.

collaborators
Showing cs.CLShow all

1 paper · 1 filter

cs.CL2026

In-Place Tokenizer Expansion for Pre-trained LLMs

Jimmy T. H. Smith, Tarek Dakhran, Alberto Cabrera +7

The paper proposes an in‑place tokenizer expansion method that continues a pre‑trained model’s BPE merges on multilingual data, reuses existing token embeddings, and initializes ne…