5 papers · 1 filter
Spectral Alignment as Predictor of Loss Explosion in Neural Network Training
Haiquan Qiu, You Wu, Yingjie Tan +2
Loss explosions in training deep neural networks can nullify multi-million dollar training runs. Conventional monitoring metrics like weight and gradient norms are often lagging an…
Why Low-Precision Transformer Training Fails: An Analysis on Flash Attention
Haiquan Qiu, Quanming Yao
The pursuit of computational efficiency has driven the adoption of low-precision formats for training transformer models. However, this progress is often hindered by notorious trai…
Superpose Task-specific Features for Model Merging
Haiquan Qiu, You Wu, Dong Li +2
Model merging enables powerful capabilities in neural networks without requiring additional training. In this paper, we introduce a novel perspective on model merging by leveraging…
Neural Symbolic Regression of Complex Network Dynamics
Haiquan Qiu, Shuzhi Liu, Quanming Yao
Complex networks describe important structures in nature and society, composed of nodes and the edges that connect them. The evolution of these networks is typically described by d…
Graph Unitary Message Passing
Haiquan Qiu, Quanming Yao
Unitarity is a useful principle for stabilizing deep neural networks, but in graph neural networks (GNNs) instability is induced not only by learnable parameters but also by the gr…