37 citations · 63 across the 22 of their papers we have counts for
6 papers · 1 filter
Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers
Vukasin Bozic, Danilo Dordevic, Daniele Coppola +2
This work presents an analysis of the effectiveness of using standard shallow feed-forward networks to mimic the behavior of the attention mechanism in the original Transformer mod…
Towards guarantees for parameter isolation in continual learning
Giulia Lanzillotta, Sidak Pal Singh, Benjamin F. Grewe +1
Deep learning has proved to be a successful paradigm for solving many challenges in machine learning. However, deep neural networks fail when trained sequentially on multiple tasks…
Transformer Fusion with Optimal Transport
Moritz Imfeld, Jacopo Graldi, Marco Giordano +3
Fusion is a technique for merging multiple independently-trained neural networks in order to combine their capabilities. Past attempts have been restricted to the case of fully-con…
On the curvature of the loss landscape
Alison Pouplin, Hrittik Roy, Sidak Pal Singh +1
One of the main challenges in modern deep learning is to understand why such over-parameterized models perform so well when trained on finite data. A way to analyze this generaliza…
The Hessian perspective into the Nature of Convolutional Neural Networks
Sidak Pal Singh, Thomas Hofmann, Bernhard Schölkopf
While Convolutional Neural Networks (CNNs) have long been investigated and applied, as well as theorized, we aim to provide a slightly different perspective into their nature -- th…
Some Fundamental Aspects about Lipschitz Continuity of Neural Networks
Grigory Khromov, Sidak Pal Singh
Lipschitz continuity is a crucial functional property of any predictive model, that naturally governs its robustness, generalisation, as well as adversarial vulnerability. Contrary…