353 citations · 1.2k across the 32 of their papers we have counts for
3 papers · 1 filter
h-detach: Modifying the LSTM Gradient Towards Better Optimization
Devansh Arpit, Bhargav Kanuparthi, Giancarlo Kerg +3
Recurrent neural networks are known for their notorious exploding and vanishing gradient problem (EVGP). This problem becomes more evident in tasks where the information needed to…
On the Spectral Bias of Neural Networks
Nasim Rahaman, Aristide Baratin, Devansh Arpit +5
Neural networks are known to be a class of highly expressive functions able to fit even random input-output mappings with accuracy. In this work, we present properties of n…
A Walk with SGD
Chen Xing, Devansh Arpit, Christos Tsirigotis +1
We present novel empirical observations regarding how stochastic gradient descent (SGD) navigates the loss landscape of over-parametrized deep neural networks (DNNs). These observa…