108 citations · 222 across the 5 of their papers we have counts for
4 papers · 1 filter
On the asymptotics of wide networks with polynomial activations
Kyle Aitken, Guy Gur-Ari
We consider an existing conjecture addressing the asymptotic behavior of neural networks in the large width limit. The results that follow from this conjecture include tight bounds…
Wider Networks Learn Better Features
Dar Gilboa, Guy Gur-Ari
Transferability of learned features between tasks can massively reduce the cost of training a neural network on a novel task. We investigate the effect of network width on learned…
Asymptotics of Wide Networks from Feynman Diagrams
Ethan Dyer, Guy Gur-Ari
Understanding the asymptotic behavior of wide networks is of considerable interest. In this work, we present a general method for analyzing this large width behavior. The method is…
Gradient Descent Happens in a Tiny Subspace
Guy Gur-Ari, Daniel A. Roberts, Ethan Dyer
We show that in a variety of large-scale deep learning scenarios the gradient dynamically converges to a very small subspace after a short period of training. The subspace is spann…