19 citations · 66 across the 7 of their papers we have counts for
7 papers
On Neural Architecture Inductive Biases for Relational Tasks
Giancarlo Kerg, Sarthak Mittal, David Rolnick +3
Current deep learning approaches have shown good in-distribution generalization performance, but struggle with out-of-distribution generalization. This is especially true in the ca…
Continuous-Time Meta-Learning with Forward Mode Differentiation
Tristan Deleu, David Kanaa, Leo Feng +4
Drawing inspiration from gradient-based meta-learning methods with infinitely small gradient steps, we introduce Continuous-Time Meta-Learning (COMLN), a meta-learning algorithm wh…
Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalization
Stanislaw Jastrzebski, Devansh Arpit, Oliver Astrand +6
The early phase of training a deep neural network has a dramatic effect on the local curvature of the loss function. For instance, using a small learning rate does not guarantee st…
Advantages of biologically-inspired adaptive neural activation in RNNs during learning
Victor Geadah, Giancarlo Kerg, Stefan Horoi +2
Dynamic adaptation in single-neuron response plays a fundamental role in neural coding in biological neural networks. Yet, most neural activation functions used in artificial netwo…
Untangling tradeoffs between recurrence and self-attention in neural networks
Giancarlo Kerg, Bhargav Kanuparthi, Anirudh Goyal +3
Attention and self-attention mechanisms, are now central to state-of-the-art deep learning on sequential tasks. However, most recent progress hinges on heuristic approaches with li…
Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics
Giancarlo Kerg, Kyle Goyette, Maximilian Puelma Touzel +4
A recent strategy to circumvent the exploding and vanishing gradient problem in RNNs, and to allow the stable propagation of signals over long time scales, is to constrain recurren…
h-detach: Modifying the LSTM Gradient Towards Better Optimization
Devansh Arpit, Bhargav Kanuparthi, Giancarlo Kerg +3
Recurrent neural networks are known for their notorious exploding and vanishing gradient problem (EVGP). This problem becomes more evident in tasks where the information needed to…