3 papers
cs.CL2026
Qwenfant: Train Big Character-level BabyLMs on Concise Parallel Text
Lukas Edman, Alexander Fraser
This work concerns our submission for the Multilingual track of the 2026 BabyLM Challenge. We improve on our baseline using data that has been summarized and translated by an LLM,…
cs.CL2026
Squeezing More from Limited Data with Recursive Transformers
Serdar Gülbahar, Lukas Edman, Alexander Fraser
Pre-training under limited data requires a different view of scaling than web-scale language modeling. With a fixed data budget but relatively abundant compute, increasing paramete…
cs.CL2025
Mask and You Shall Receive: Optimizing Masked Language Modeling For Pretraining BabyLMs
Lukas Edman, Alexander Fraser
We describe our strategy for the 2025 edition of the BabyLM Challenge. Our main contribution is that of an improved form of Masked Language Modeling (MLM), which adapts the probabi…