14.5k citations · 90.1k across the 347 of their papers we have counts for
14 papers · 2 filters
Depth with Nonlinearity Creates No Bad Local Minima in ResNets
Kenji Kawaguchi, Yoshua Bengio
In this paper, we prove that depth with nonlinearity creates no bad local minima in a type of arbitrarily deep ResNets with arbitrary nonlinear activation functions, in the sense t…
h-detach: Modifying the LSTM Gradient Towards Better Optimization
Devansh Arpit, Bhargav Kanuparthi, Giancarlo Kerg +3
Recurrent neural networks are known for their notorious exploding and vanishing gradient problem (EVGP). This problem becomes more evident in tasks where the information needed to…
Deep Graph Infomax
Petar Veličković, William Fedus, William L. Hamilton +3
We present Deep Graph Infomax (DGI), a general approach for learning node representations within graph-structured data in an unsupervised manner. DGI relies on maximizing mutual in…
Learning deep representations by mutual information estimation and maximization
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon +4
In this work, we perform unsupervised learning of representations by maximizing mutual information between an input and the output of a deep neural network encoder. Importantly, we…
On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length
Stanisław Jastrzębski, Zachary Kenton, Nicolas Ballas +3
Stochastic Gradient Descent (SGD) based training of neural networks with a large learning rate or a small batch-size typically ends in well-generalizing, flat regions of the weight…
Focused Hierarchical RNNs for Conditional Sequence Processing
Nan Rosemary Ke, Konrad Zolna, Alessandro Sordoni +6
Recurrent Neural Networks (RNNs) with attention mechanisms have obtained state-of-the-art results for many sequence processing tasks. Most of these models use a simple form of enco…