In-Place Tokenizer Expansion for Pre-trained LLMs

Jimmy T. H. Smith, Tarek Dakhran, Alberto Cabrera +7

A tokenizer fixed at the start of pre-training allocates vocabulary in proportion to the pre-training corpus, reflecting the deployment priorities at that time. When those prioriti…