551 citations · 2.1k across the 36 of their papers we have counts for
4 papers · 1 filter
Disentangling Trainability and Generalization in Deep Neural Networks
Lechao Xiao, Jeffrey Pennington, Samuel S. Schoenholz
A longstanding goal in the theory of deep learning is to characterize the conditions under which a given neural network architecture will be trainable, and if so, how well it might…
A Random Matrix Perspective on Mixtures of Nonlinearities for Deep Learning
Ben Adlam, Jake Levinson, Jeffrey Pennington
One of the distinguishing characteristics of modern deep learning systems is that they typically employ neural network architectures that utilize enormous numbers of parameters, of…
A Mean Field Theory of Batch Normalization
Greg Yang, Jeffrey Pennington, Vinay Rao +2
We develop a mean field theory for batch normalization in fully-connected feedforward neural networks. In so doing, we provide a precise characterization of signal propagation and…
Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs
Dar Gilboa, Bo Chang, Minmin Chen +4
Training recurrent neural networks (RNNs) on long sequence tasks is plagued with difficulties arising from the exponential explosion or vanishing of signals as they propagate forwa…