2 citations · 5 across the 8 of their papers we have counts for
5 papers · 1 filter
DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data
Zhen Huang, Yikun Wang, Shijie Xia +1
Pretraining data processing is critical to the downstream performance of Large Language Models (LLMs). However, many existing approaches define a fixed processing strategy at the c…
UMoE:Unlocking Every Expert in Domain-Specific Training
Xuefeng Li, Pengfei Liu
Mixture-of-Experts (MoE) models scale capacity without proportional compute cost and have become a key architecture for frontier large language models (LLMs). Yet domain-specific p…
Generative AI Act II: Test Time Scaling Drives Cognition Engineering
Shijie Xia, Yiwei Qin, Xuefeng Li +11
The first generation of Large Language Models - what might be called "Act I" of generative AI (2020-2023) - achieved remarkable success through massive parameter and data scaling,…
ToRL: Scaling Tool-Integrated RL
Xuefeng Li, Haoyang Zou, Pengfei Liu
We introduce ToRL (Tool-Integrated Reinforcement Learning), a framework for training large language models (LLMs) to autonomously use computational tools via reinforcement learning…
O1 Replication Journey -- Part 2: Surpassing O1-preview through Simple Distillation, Big Progress or Bitter Lesson?
Zhen Huang, Haoyang Zou, Xuefeng Li +7
This paper presents a critical examination of current approaches to replicating OpenAI's O1 model capabilities, with particular focus on the widespread but often undisclosed use of…