68 citations · 160 across the 28 of their papers we have counts for
6 papers · 1 filter
Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networks
Greg Yang, Dingli Yu, Chen Zhu +1
By classifying infinite-width neural networks and identifying the *optimal* limit, Tensor Programs IV and V demonstrated a universal way, called P, for *widthwise hyperparameter…
Commutative Width and Depth Scaling in Deep Neural Networks
Soufiane Hayou
This paper is the second in the series Commutative Scaling of Width and Depth (WD) about commutativity of infinite width and depth limits in deep neural networks. Our aim is to und…
On the Connection Between Riemann Hypothesis and a Special Class of Neural Networks
Soufiane Hayou
The Riemann hypothesis (RH) is a long-standing open problem in mathematics. It conjectures that non-trivial zeros of the zeta function all have real part equal to 1/2. The extent o…
Leave-one-out Distinguishability in Machine Learning
Jiayuan Ye, Anastasia Borovykh, Soufiane Hayou +1
We introduce an analytical framework to quantify the changes in a machine learning algorithm's output distribution following the inclusion of a few data points in its training set,…
Data pruning and neural scaling laws: fundamental limitations of score-based algorithms
Fadhel Ayed, Soufiane Hayou
Data pruning algorithms are commonly used to reduce the memory and computational cost of the optimization process. Recent empirical results reveal that random data pruning remains…
Width and Depth Limits Commute in Residual Networks
Soufiane Hayou, Greg Yang
We show that taking the width and depth to infinity in a deep neural network with skip connections, when branches are scaled by (the only nontrivial scaling), resu…