DVAGen: Dynamic Vocabulary Augmented Generation

Wei Du, Nuowei Liu, Jie Wang +4

Language models trained with a fixed vocabulary struggle to generalize to novel or out-of-vocabulary words, limiting their flexibility in handling diverse token combinations. Exist…