168 citations · 390 across the 9 of their papers we have counts for
Showing 2020Show all
3 papers · 1 filter
cs.LG2020★ 8 cited
On the asymptotics of wide networks with polynomial activations
Kyle Aitken, Guy Gur-Ari
We consider an existing conjecture addressing the asymptotic behavior of neural networks in the large width limit. The results that follow from this conjecture include tight bounds…
stat.ML2020
On the training dynamics of deep networks with regularization
Aitor Lewkowycz, Guy Gur-Ari
We study the role of regularization in deep learning, and uncover simple relations between the performance of the model, the coefficient, the learning rate, and the num…
stat.ML2020★ 58 cited
The large learning rate phase of deep learning: the catapult mechanism
Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer +2
The choice of initial learning rate can have a profound effect on the performance of deep networks. We present a class of neural networks with solvable training dynamics, and confi…