Scaling Domain Data Repetition in LLM Pretraining

Jingwei Li, Xinran Gu, Rui Dai +5

As large language models scale, their training-token budgets must also increase to maintain an appropriate tokens-per-parameter ratio (\(\mathrm{TPP}\)). However, high-quality doma…