3.7k citations · 10.2k across the 30 of their papers we have counts for
Showing 2023 · cs.CLShow all
2 papers · 2 filters
cs.CL2023★ 7 cited
A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity
Shayne Longpre, Gregory Yauney, Emily Reif +8
Pretraining is the preliminary and fundamental step in developing capable language models (LM). Despite this, pretraining data design is critically under-documented and often guide…
cs.CL2023★ 6 cited
UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretraining
Hyung Won Chung, Noah Constant, Xavier Garcia +4
Pretrained multilingual large language models have typically used heuristic temperature-based sampling to balance between different languages. However previous work has not systema…