5 papers
HetCCL: Enabling Collective Communication For Mixed-Vendor Heterogeneous Clusters
Yuejie Wang, Tao Chang, Yuanyuan Zhao +10
Training Large Language Models (LLMs) on heterogeneous clusters presents significant challenges for collective communication, as hardware from multiple vendors introduces diverse n…
DataFlex: A Unified Framework for Data-Centric Dynamic Training of Large Language Models
Hao Liang, Zhengyang Zhao, Meiyi Qiang +22
Data-centric training has emerged as a promising direction for improving large language models (LLMs) by optimizing not only model parameters but also the selection, composition, a…
Towards Next-Generation LLM Training: From the Data-Centric Perspective
Hao Liang, Zhengyang Zhao, Zhaoyang Han +8
Large language models (LLMs) have demonstrated remarkable performance across a wide range of tasks and domains, with data playing a central role in enabling these advances. Despite…
PonderLM: Pretraining Language Models to Ponder in Continuous Space
Boyi Zeng, Shixiang Song, Siyuan Huang +6
Humans ponder before articulating complex sentence elements, enabling deeper cognitive processing through focused effort. In this work, we introduce this pondering process into lan…
Innovator: Scientific Continued Pretraining with Fine-grained MoE Upcycling
Ning Liao, Xiaoxing Wang, Zehao Lin +18
A large language model (LLM) with knowledge in both scientific and general tasks is the foundation of science general intelligence. However, directly continued pretraining an LLM u…