20 papers
DD-RNO: A Domain-Decomposed Routed Neural Operator for Airfoil Flow Prediction
T. A. Mehta, P. S. Bhati, H. D. Akolekar
Deep learning surrogates for RANS flow prediction around airfoils face two persistent bottlenecks. A single neural architecture cannot simultaneously resolve sharp near-wall bounda…
Scalable Visual Pretraining for Language Intelligence
Yiming Zhang, Zhonghan Zhao, Wenwei Zhang +14
The rapid progress of large foundation models has been driven predominantly by pretraining on large-scale text corpora. However, many forms of knowledge are conveyed through visual…
Which Reasoning Trajectories Teach Students to Reason Better? A Simple Metric of Informative Alignment
Yuming Yang, Mingyoung Lai, Wanxu Zhao +13
Long chain-of-thought (CoT) trajectories provide rich supervision signals for distilling reasoning from teacher to student LLMs. However, both prior work and our experiments show t…
Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data
Xu Guo, Runyu Peng, Jian Tong +4
Large language models (LLMs) rely on web-scale corpora for pre-training. The noise inherent in these datasets tends to obscure meaningful patterns and ultimately degrade model perf…
Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioning
Yechen Zhang, Shuhao Xing, Junhao Huang +5
Recent advances in spectral optimization, notably Muon, have demonstrated that constraining update steps to the Stiefel manifold can significantly accelerate training and improve g…
Evolution of Concepts in Language Model Pre-Training
Xuyang Ge, Wentao Shu, Jiaxing Wu +3
Language models obtain extensive capabilities through pre-training. However, the pre-training process remains a black box. In this work, we track linear interpretable feature evolu…