Selecting Repetition Counts Across Model Scales in Data-Constrained Pretraining

Ziyue WANG, T. Kanamori

The repetition count that works best for a small language model may not remain best at a larger scale. We study this effect in pretraining with a finite target corpus mixed with ge…