TokAlign++: Advancing Vocabulary Adaptation via Better Token Alignment

Chong Li, Yingzhuo Deng, Wen Yang +2

Tokenization is a foundational step in the text process of Large Language Models (LLMs). Texts must be first tokenized into token IDs, which are then input to LLMs. Inefficient tok…