48 citations · 121 across the 23 of their papers we have counts for
Showing 2026Show all
2 papers · 1 filter
cs.CL2026
Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior
Atsuki Yamaguchi, Tatsuro Inaba, Joel Niklaus +3
Pre-pretraining (PPT) on synthetic non-natural language data improves token efficiency during language model pre-training (PT). Prior work attributes this gain to a grammatical pri…
cs.CL2026
How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data
Joel Niklaus, Atsuki Yamaguchi, Michal Štefánik +9
Synthetic data is a standard component in training large language models, yet systematic comparisons across design dimensions, including rephrasing strategy, generator model, and s…