Different Corpora, Different Skills? A Paired Study of Small Language Models

Edoardo Vaira, Georgios Chavales, Pierre Nugues

Small language models are often trained under tight data budgets, where the choice of pretraining corpus may matter as much as model size or training recipe. We compare two contras…