Training Bilingual LMs with Data Constraints in the Targeted Language

Skyler Seto, Maartje ter Hoeve, Richard He Bai +2

Large language models are trained on massive scrapes of the web, as required by current scaling laws. Most progress is made for English, given its abundance of high-quality pretrai…