1 citations · 1 across the 15 of their papers we have counts for
3 papers · 1 filter
LaCy: What Small Language Models Can and Should Learn is Not Just a Question of Loss
Szilvia Ujváry, Louis Béthune, Pierre Ablin +3
Language models have consistently grown to compress more world knowledge into their parameters, but the knowledge that can be pretrained into them is upper-bounded by their paramet…
Optimal Splitting of Language Models from Mixtures to Specialized Domains
Skyler Seto, Pierre Ablin, Anastasiia Filippova +4
Language models achieve impressive performance on a variety of knowledge, language, and reasoning tasks due to the scale and diversity of pretraining data available. The standard t…
Task-Adaptive Pretrained Language Models via Clustered-Importance Sampling
David Grangier, Simin Fan, Skyler Seto +1
Specialist language models (LMs) focus on a specific task or domain on which they often outperform generalist LMs of the same size. However, the specialist data needed to pretrain…