551 citations · 2.1k across the 36 of their papers we have counts for
3 papers · 1 filter
Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice
Jeffrey Pennington, Samuel S. Schoenholz, Surya Ganguli
It is well known that the initialization of weights in deep neural networks can have a dramatic impact on learning speed. For example, ensuring the mean squared singular value of a…
Deep Neural Networks as Gaussian Processes
Jaehoon Lee, Yasaman Bahri, Roman Novak +3
It has long been known that a single-layer fully-connected neural network with an i.i.d. prior over its parameters is equivalent to a Gaussian process (GP), in the limit of infinit…
A Correspondence Between Random Neural Networks and Statistical Field Theory
Samuel S. Schoenholz, Jeffrey Pennington, Jascha Sohl-Dickstein
A number of recent papers have provided evidence that practical design questions about neural networks may be tackled theoretically by studying the behavior of random networks. How…