978 citations · 1.6k across the 24 of their papers we have counts for
3 papers · 1 filter
Tensor Switching Networks
Chuan-Yung Tsai, Andrew Saxe, David Cox
We present a novel neural network algorithm, the Tensor Switching (TS) network, which generalizes the Rectified Linear Unit (ReLU) nonlinearity to tensor-valued hidden units. The T…
Qualitatively characterizing neural network optimization problems
Ian J. Goodfellow, Oriol Vinyals, Andrew M. Saxe
Training neural networks involves solving large-scale non-convex optimization problems. This task has long been believed to be extremely difficult, with fear of local minima and ot…
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
Andrew M. Saxe, James L. McClelland, Surya Ganguli
Despite the widespread practical success of deep learning methods, our theoretical understanding of the dynamics of learning in deep neural networks remains quite sparse. We attemp…