Self-Play Pretraining with Zero Data

Aditya Cowsik, Kfir Dolev, Michael Y. Li +4

Advances in language modeling have been driven by scaling pretraining on ever more data. Yet, the training data is still largely curated on the model's behalf. A more general appro…