7 papers
Why Post-Norm Transformers Collapse: Attention Amplification and Gradient Repair Failure
Xingjian Wang, Qingyu Han, Xiaodong Luo +1
Deep decoder-only Transformers often replace the original Post-Norm architecture with Pre-Norm variants because Post-Norm training is highly sensitive to warmup and learning rate u…
SLAI T-Rex: Full-Parameter Post-training of the DeepSeek-V4 Family on Ascend SuperPOD
Dongfang Li, Xiaodong Luo, Ruoyu Sun +64
Full-parameter post-training of trillion-parameter-scale MoE models introduces substantial system-level challenges for large-scale distributed training, including severe memory pre…
QuadEnhancer: Leveraging Quadratic Transformations to Enhance Deep Neural Networks
Qian Chen, Linxin Yang, Akang Wang +2
The combination of linear transformations and non-linear activation functions forms the foundation of most modern deep neural networks, enabling them to approximate highly complex…
Solving Quadratic Programs via Deep Unrolled Douglas-Rachford Splitting
Jinxin Xiong, Xi Gao, Linxin Yang +3
Convex quadratic programs (QPs) are fundamental to numerous applications, including finance, engineering, and energy systems. Among the various methods for solving them, the Dougla…
When GNNs meet symmetry in ILPs: an orbit-based feature augmentation approach
Qian Chen, Lei Li, Qian Li +6
A common characteristic in integer linear programs (ILPs) is symmetry, allowing variables to be permuted without altering the underlying problem structure. Recently, GNNs have emer…
SymILO: A Symmetry-Aware Learning Framework for Integer Linear Optimization
Qian Chen, Tianjian Zhang, Linxin Yang +5
Integer linear programs (ILPs) are commonly employed to model diverse practical problems such as scheduling and planning. Recently, machine learning techniques have been utilized t…