14 citations · 31 across the 27 of their papers we have counts for
3 papers · 1 filter
How to Set the Batch Size for Large-Scale Pre-training?
Yunhua Zhou, Junhao Huang, Shuhao Xing +4
The concept of Critical Batch Size, as pioneered by OpenAI, has long served as a foundational principle for large-scale pre-training. However, with the paradigm shift towards the W…
How to Set the Learning Rate for Large-Scale Pre-training?
Yunhua Zhou, Shuhao Xing, Junhao Huang +2
Optimal configuration of the learning rate (LR) is a fundamental yet formidable challenge in large-scale pre-training. Given the stringent trade-off between training costs and mode…
Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspective
Zhiyuan Zeng, Qinyuan Cheng, Zhangyue Yin +6
OpenAI o1 represents a significant milestone in Artificial Inteiligence, which achieves expert-level performances on many challanging tasks that require strong reasoning ability.Op…