5 papers
Exploring Polyglot Harmony: On Multilingual Data Allocation for Large Language Models Pretraining
Ping Guo, Yubing Ren, Binbin Liu +6
Large language models (LLMs) have become integral to a wide range of applications worldwide, driving an unprecedented global demand for effective multilingual capabilities. Central…
TiKMiX: Take Data Influence into Dynamic Mixture for Language Model Pre-training
Yifan Wang, Binbin Liu, Fengze Liu +6
The data mixture used in the pre-training of a language model is a cornerstone of its final performance. However, a static mixing strategy is suboptimal, as the model's learning pr…
MuBench: Assessment of Multilingual Capabilities of Large Language Models Across 61 Languages
Wenhan Han, Yifan Zhang, Zhixun Chen +7
Multilingual large language models (LLMs) are advancing rapidly, with new models frequently claiming support for an increasing number of languages. However, existing evaluation dat…
MoORE: SVD-based Model MoE-ization for Conflict- and Oblivion-Resistant Multi-Task Adaptation
Shen Yuan, Yin Zheng, Taifeng Wang +2
Adapting large-scale foundation models in multi-task scenarios often suffers from task conflict and oblivion. To mitigate such issues, we propose a novel ''model MoE-ization'' stra…
QuaDMix: Quality-Diversity Balanced Data Selection for Efficient LLM Pretraining
Fengze Liu, Weidong Zhou, Binbin Liu +8
Quality and diversity are two critical metrics for the training data of large language models (LLMs), positively impacting performance. Existing studies often optimize these metric…