4 papers
TokAlign++: Advancing Vocabulary Adaptation via Better Token Alignment
Chong Li, Yingzhuo Deng, Wen Yang +2
Tokenization is a foundational step in the text process of Large Language Models (LLMs). Texts must be first tokenized into token IDs, which are then input to LLMs. Inefficient tok…
Parallel Scaling Law: Unveiling Reasoning Generalization through A Cross-Linguistic Perspective
Wen Yang, Junhong Wu, Chong Li +2
Recent advancements in Reinforcement Post-Training (RPT) have significantly enhanced the capabilities of Large Reasoning Models (LRMs), sparking increased interest in the generaliz…
Group then Scale: Dynamic Mixture-of-Experts Multilingual Language Model
Chong Li, Yingzhuo Deng, Jiajun Zhang +1
The curse of multilinguality phenomenon is a fundamental problem of multilingual Large Language Models (LLMs), where the competition between massive languages results in inferior p…
TokAlign: Efficient Vocabulary Adaptation via Token Alignment
Chong Li, Jiajun Zhang, Chengqing Zong
Tokenization serves as a foundational step for Large Language Models (LLMs) to process text. In new domains or languages, the inefficiency of the tokenizer will slow down the train…