11 papers
Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipe
Qian Zhao, Kunlong Chen, Changxin Tian +9
FP4 training promises substantial reductions in memory and computation cost for LLM pretraining, yet current FP4 hardware paths and recipes, including NVIDIA Blackwell/Rubin-class…
Ling and Ring 2.6 Technical Report: Efficient and Instant Agentic Intelligence at Trillion-Parameter Scale
Ang Li, Ben Liu, Bin Han +215
Efficient and scalable agentic intelligence requires models that can deliver both low-latency responses and strong reasoning capabilities while remaining practical to train, serve,…
SuperValid: Capability-Aligned OOD Validation for Generalizable Downstream Scaling
Quanen Sun, Changxin Tian, Ke Shi +5
Scaling laws guide large language model training by relating compute to cross-entropy loss, and recent work further extends them to predict downstream benchmark performance. Howeve…
PowLU: An Activation Function for Stable Pre-Training of LLMs
Peijie Jiang, Yuqi Feng, Cunyin Peng +5
In contemporary large language models (LLMs), the swish-gated linear unit (SwiGLU) activation function is widely adopted to regulate the information flow and introduce non-linearit…
MaP: A Unified Framework for Reliable Evaluation of Pre-training Dynamics
Jiapeng Wang, Changxin Tian, Kunlong Chen +5
Reliable evaluation is fundamental to the progress of Large Language Models (LLMs), yet the evaluation process during pre-training is plagued by significant instability that obscur…
MergeMix: Optimizing Mid-Training Data Mixtures via Learnable Model Merging
Jiapeng Wang, Changxin Tian, Kunlong Chen +5
Optimizing data mixtures is essential for unlocking the full potential of large language models (LLMs), yet identifying the optimal composition remains computationally prohibitive…