2 citations · 2 across the 5 of their papers we have counts for
4 papers · 1 filter
Training Dynamics of Transformers to Recognize Word Co-occurrence via Gradient Flow Analysis
Hongru Yang, Bhavya Kailkhura, Zhangyang Wang +1
Understanding the training dynamics of transformers is important to explain the impressive capabilities behind large language models. In this work, we study the dynamics of trainin…
Pruning Before Training May Improve Generalization, Provably
Hongru Yang, Yingbin Liang, Xiaojie Guo +2
It has been observed in practice that applying pruning-at-initialization methods to neural networks and training the sparsified networks can not only retain the testing performance…
Neural Networks with Sparse Activation Induced by Large Bias: Tighter Analysis with Bias-Generalized NTK
Hongru Yang, Ziyu Jiang, Ruizhe Zhang +2
We study training one-hidden-layer ReLU networks in the neural tangent kernel (NTK) regime, where the networks' biases are initialized to some constant rather than zero. We prove t…
On the Neural Tangent Kernel Analysis of Randomly Pruned Neural Networks
Hongru Yang, Zhangyang Wang
Motivated by both theory and practice, we study how random pruning of the weights affects a neural network's neural tangent kernel (NTK). In particular, this work establishes an eq…