281 citations · 451 across the 25 of their papers we have counts for
3 papers · 1 filter
Improving Baselines in the Wild
Kazuki Irie, Imanol Schlag, Róbert Csordás +1
We share our experience with the recently released WILDS benchmark, a collection of ten datasets dedicated to developing models and training strategies which are robust to domain s…
Going Beyond Linear Transformers with Recurrent Fast Weight Programmers
Kazuki Irie, Imanol Schlag, Róbert Csordás +1
Transformers with linearised attention (''linear Transformers'') have demonstrated the practical scalability and effectiveness of outer product-based Fast Weight Programmers (FWPs)…
Linear Transformers Are Secretly Fast Weight Programmers
Imanol Schlag, Kazuki Irie, Jürgen Schmidhuber
We show the formal equivalence of linearised self-attention mechanisms and fast weight controllers from the early '90s, where a ``slow" neural net learns by gradient descent to pro…