551 citations · 2.1k across the 36 of their papers we have counts for
6 papers · 1 filter
Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
Roman Novak, Lechao Xiao, Jaehoon Lee +6
There is a previously identified equivalence between wide fully connected neural networks (FCNs) and Gaussian processes (GPs). This equivalence enables, for instance, test set pred…
Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein +2
In recent years, state-of-the-art methods in computer vision have utilized increasingly deep convolutional neural network architectures (CNNs), with some of the most successful mod…
Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks
Minmin Chen, Jeffrey Pennington, Samuel S. Schoenholz
Recurrent neural networks have gained widespread use in modeling sequence data across various domains. While many successful recurrent architectures employ a notion of gating, the…
The Emergence of Spectral Universality in Deep Networks
Jeffrey Pennington, Samuel S. Schoenholz, Surya Ganguli
Recent work has shown that tight concentration of the entire spectrum of singular values of a deep network's input-output Jacobian around one at initialization can speed up learnin…
Estimating the Spectral Density of Large Implicit Matrices
Ryan P. Adams, Jeffrey Pennington, Matthew J. Johnson +4
Many important problems are characterized by the eigenvalues of a large matrix. For example, the difficulty of many optimization problems, such as those arising from the fitting of…
Sensitivity and Generalization in Neural Networks: an Empirical Study
Roman Novak, Yasaman Bahri, Daniel A. Abolafia +2
In practice it is often found that large over-parameterized neural networks generalize better than their smaller counterparts, an observation that appears to conflict with classica…