18.2k citations
- University of Applied Sciences and Arts of Southern SwitzerlandCH65 papers
- Università della Svizzera italianaCH39 papers
- ETH ZurichCH9 papers
- Data61AU8 papers
- Politecnico di MilanoIT7 papers
- University of BolognaIT7 papers
- Centre National de la Recherche ScientifiqueFR4 papers
- Ghent UniversityBE3 papers
- Google (United States)US3 papers
- Hokkaido UniversityJP3 papers
- IMT School for Advanced Studies LuccaIT3 papers
- King Abdullah University of Science and TechnologySA3 papers
Showing 2021 · cs.LGShow all
3 papers · 2 filters
cs.LG2021★ 4 cited
The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization
Róbert Csordás, Kazuki Irie, Jürgen Schmidhuber
Despite progress across a broad range of applications, Transformers have limited success in systematic generalization. The situation is especially frustrating in the case of algori…
cs.LG2021★ 5 cited
The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers
Róbert Csordás, Kazuki Irie, Jürgen Schmidhuber
Recently, many datasets have been proposed to test the systematic generalization ability of neural networks. The companion baseline Transformers, typically trained with default hyp…
cs.LG2021★ 24 cited
Linear Transformers Are Secretly Fast Weight Programmers
Imanol Schlag, Kazuki Irie, Jürgen Schmidhuber
We show the formal equivalence of linearised self-attention mechanisms and fast weight controllers from the early '90s, where a ``slow" neural net learns by gradient descent to pro…