2 papers
cs.LG2020
Untangling tradeoffs between recurrence and self-attention in neural networks
Giancarlo Kerg, Bhargav Kanuparthi, Anirudh Goyal +3
Attention and self-attention mechanisms, are now central to state-of-the-art deep learning on sequential tasks. However, most recent progress hinges on heuristic approaches with li…
stat.ML2018
h-detach: Modifying the LSTM Gradient Towards Better Optimization
Devansh Arpit, Bhargav Kanuparthi, Giancarlo Kerg +3
Recurrent neural networks are known for their notorious exploding and vanishing gradient problem (EVGP). This problem becomes more evident in tasks where the information needed to…