FLEXITOKENS: Flexible Tokenization for Evolving Language Models

Abraham Toluwase Owodunni, Orevaoghene Ahia, Sachin Kumar

Adapting language models to new data distributions by simple finetuning is challenging. This is due to the rigidity of their subword tokenizers, which typically remain unchanged du…