1.4k citations · 7.9k across the 69 of their papers we have counts for
Showing 2017 · cs.LGShow all
2 papers · 2 filters
cs.LG2017★ 69 cited
Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice
Jeffrey Pennington, Samuel S. Schoenholz, Surya Ganguli
It is well known that the initialization of weights in deep neural networks can have a dramatic impact on learning speed. For example, ensuring the mean squared singular value of a…
cs.LG2017★ 1k cited
Continual Learning Through Synaptic Intelligence
Friedemann Zenke, Ben Poole, Surya Ganguli
While deep learning has led to remarkable advances across diverse applications, it struggles in domains where the data distribution changes over the course of learning. In stark co…