282 citations · 917 across the 32 of their papers we have counts for
3 papers · 1 filter
Overcoming classic challenges for artificial neural networks by providing incentives and practice
Kazuki Irie, Brenden M. Lake
Since the earliest proposals for artificial neural network (ANN) models of the mind and brain, critics have pointed out key weaknesses in these models compared to human cognitive a…
MoEUT: Mixture-of-Experts Universal Transformers
Róbert Csordás, Kazuki Irie, Jürgen Schmidhuber +2
Previous work on Universal Transformers (UTs) has demonstrated the importance of parameter sharing across layers. By allowing recurrence in depth, UTs have advantages over standard…
Dissecting the Interplay of Attention Paths in a Statistical Mechanics Theory of Transformers
Lorenzo Tiberi, Francesca Mignacco, Kazuki Irie +1
Despite the remarkable empirical performance of Transformers, their theoretical understanding remains elusive. Here, we consider a deep multi-head self-attention network, that is c…