3 papers
cs.AI2026
Scaling Domain Data Repetition in LLM Pretraining
Jingwei Li, Xinran Gu, Rui Dai +5
As large language models scale, their training-token budgets must also increase to maintain an appropriate tokens-per-parameter ratio (\(\mathrm{TPP}\)). However, high-quality doma…
cs.LG2026
Explaining Data Mixing Scaling Laws
Rui Dai, Shuran Zheng
Recent research has established empirical scaling laws to predict model performance on multi-domain data mixtures. However, a theoretical understanding of these model loss behavior…
cs.LG2025
Leveraging Submodule Linearity Enhances Task Arithmetic Performance in LLMs
Rui Dai, Sile Hu, Xu Shen +3
Task arithmetic is a straightforward yet highly effective strategy for model merging, enabling the resultant model to exhibit multi-task capabilities. Recent research indicates tha…