282 citations · 917 across the 32 of their papers we have counts for
6 papers · 1 filter
Improving Baselines in the Wild
Kazuki Irie, Imanol Schlag, Róbert Csordás +1
We share our experience with the recently released WILDS benchmark, a collection of ten datasets dedicated to developing models and training strategies which are robust to domain s…
Training and Generating Neural Networks in Compressed Weight Space
Kazuki Irie, Jürgen Schmidhuber
The inputs and/or outputs of some neural nets are weight matrices of other neural nets. Indirect encodings or end-to-end compression of weight matrices could help to scale such app…
The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization
Róbert Csordás, Kazuki Irie, Jürgen Schmidhuber
Despite progress across a broad range of applications, Transformers have limited success in systematic generalization. The situation is especially frustrating in the case of algori…
The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers
Róbert Csordás, Kazuki Irie, Jürgen Schmidhuber
Recently, many datasets have been proposed to test the systematic generalization ability of neural networks. The companion baseline Transformers, typically trained with default hyp…
Going Beyond Linear Transformers with Recurrent Fast Weight Programmers
Kazuki Irie, Imanol Schlag, Róbert Csordás +1
Transformers with linearised attention (''linear Transformers'') have demonstrated the practical scalability and effectiveness of outer product-based Fast Weight Programmers (FWPs)…
Linear Transformers Are Secretly Fast Weight Programmers
Imanol Schlag, Kazuki Irie, Jürgen Schmidhuber
We show the formal equivalence of linearised self-attention mechanisms and fast weight controllers from the early '90s, where a ``slow" neural net learns by gradient descent to pro…