4 papers · 1 filter
daVinci-LLM:Towards the Science of Pretraining
Yiwei Qin, Yixiu Liu, Tiantian Mi +12
The foundational pretraining phase determines a model's capability ceiling, as post-training struggles to overcome capability foundations established during pretraining, yet it rem…
Data Darwinism Part II: DataEvolve -- AI can Autonomously Evolve Pretraining Data Curation
Tiantian Mi, Dongming Shan, Zhen Huang +6
Data Darwinism (Part I) established a ten-level hierarchy for data processing, showing that stronger processing can unlock greater data value. However, that work relied on manually…
Data Darwinism Part I: Unlocking the Value of Scientific Data for Pre-training
Yiwei Qin, Zhen Huang, Tiantian Mi +5
Data quality determines foundation model performance, yet systematic processing frameworks are lacking. We introduce Data Darwinism, a ten-level taxonomy (L0-L9) that conceptualize…
O1 Replication Journey: A Strategic Progress Report -- Part 1
Yiwei Qin, Xuefeng Li, Haoyang Zou +8
This paper introduces a pioneering approach to artificial intelligence research, embodied in our O1 Replication Journey. In response to the announcement of OpenAI's groundbreaking…