1 citations · 1 across the 11 of their papers we have counts for
4 papers · 1 filter
CCI3.0-HQ: a large-scale Chinese dataset of high quality designed for pre-training large language models
Liangdong Wang, Bo-Wen Zhang, Chengwei Wu +7
We present CCI3.0-HQ (https://huggingface.co/datasets/BAAI/CCI3-HQ), a high-quality 500GB subset of the Chinese Corpora Internet 3.0 (CCI3.0)(https://huggingface.co/datasets/BAAI/C…
Mitigating Training Imbalance in LLM Fine-Tuning via Selective Parameter Merging
Yiming Ju, Ziyi Ni, Xingrun Xing +4
Supervised fine-tuning (SFT) is crucial for adapting Large Language Models (LLMs) to specific tasks. In this work, we demonstrate that the order of training data can lead to signif…
Beyond IID: Optimizing Instruction Learning from the Perspective of Instruction Interaction and Dependency
Hanyu Zhao, Li Du, Yiming Ju +2
With the availability of various instruction datasets, a pivotal challenge is how to effectively select and integrate these instructions to fine-tune large language models (LLMs).…
AquilaMoE: Efficient Training for MoE Models with Scale-Up and Scale-Out Strategies
Bo-Wen Zhang, Liangdong Wang, Ye Yuan +24
In recent years, with the rapid application of large language models across various fields, the scale of these models has gradually increased, and the resources required for their…