7 papers
Intern-S2-Preview: Scientific Agentic Foundation Model
Lei Bai, Jiaqi Cao, Chiyu Chen +121
Scientific discovery increasingly requires AI systems that can reason over scientific evidence of heterogeneous modalities, interact with scientific tools and environments, and sus…
Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioning
Yechen Zhang, Shuhao Xing, Junhao Huang +5
Recent advances in spectral optimization, notably Muon, have demonstrated that constraining update steps to the Stiefel manifold can significantly accelerate training and improve g…
How to Set the Learning Rate for Large-Scale Pre-training?
Yunhua Zhou, Shuhao Xing, Junhao Huang +2
Optimal configuration of the learning rate (LR) is a fundamental yet formidable challenge in large-scale pre-training. Given the stringent trade-off between training costs and mode…
Pre-Trained Policy Discriminators are General Reward Models
Shihan Dou, Shichun Liu, Yuming Yang +19
We offer a novel perspective on reward modeling by formulating it as a policy discriminator, which quantifies the difference between two policies to generate a reward signal, guidi…
Capability Salience Vector: Fine-grained Alignment of Loss and Capabilities for Downstream Task Scaling Law
Qiming Ge, Shuhao Xing, Songyang Gao +8
Scaling law builds the relationship between training computation and validation loss, enabling researchers to effectively predict the loss trending of models across different level…
Code Needs Comments: Enhancing Code LLMs with Comment Augmentation
Demin Song, Honglin Guo, Yunhua Zhou +8
The programming skill is one crucial ability for Large Language Models (LLMs), necessitating a deep understanding of programming languages (PLs) and their correlation with natural…
CoLLiE: Collaborative Training of Large Language Models in an Efficient Way
Kai Lv, Shuo Zhang, Tianle Gu +11
Large language models (LLMs) are increasingly pivotal in a wide range of natural language processing tasks. Access to pre-trained models, courtesy of the open-source community, has…