24 citations · 34 across the 9 of their papers we have counts for
12 papers
Thought without systematicity? Evaluating reasoning models on rule induction tasks
Simon Schug, Brenden M. Lake
A central tenet of human cognition is systematicity, the principle that understanding one concept is inherently tied to understanding close variations of that concept. Do reasoning…
Sparsely gated tiny linear experts
Simon Schug
Sparsity allows scaling model parameters without proportionally increasing computational cost. While mixture of experts (MoE) models are made increasingly sparse, individual expert…
Are they human? Detecting large language models by probing human memory constraints
Simon Schug, Brenden M. Lake
The validity of online behavioral research relies on study participants being human rather than machine. In the past, it was possible to detect machines by posing simple challenges…
Scaling can lead to compositional generalization
Florian Redhardt, Yassir Akram, Simon Schug
Can neural networks systematically capture discrete, compositional task structure despite their continuous, distributed nature? The impressive capabilities of large-scale neural ne…
When can transformers compositionally generalize in-context?
Seijin Kobayashi, Simon Schug, Yassir Akram +5
Many tasks can be composed from a few independent components. This gives rise to a combinatorial explosion of possible tasks, only some of which might be encountered during trainin…
Attention as a Hypernetwork
Simon Schug, Seijin Kobayashi, Yassir Akram +2
Transformers can under some circumstances generalize to novel problem instances whose constituent parts might have been encountered during training, but whose compositions have not…
Discovering modular solutions that generalize compositionally
Simon Schug, Seijin Kobayashi, Yassir Akram +6
Many complex tasks can be decomposed into simpler, independent parts. Discovering such underlying compositional structure has the potential to enable compositional generalization.…
Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysis
Alexander Meulemans, Simon Schug, Seijin Kobayashi +2
To make reinforcement learning more sample efficient, we need better credit assignment methods that measure an action's influence on future rewards. Building upon Hindsight Credit…
Online learning of long-range dependencies
Nicolas Zucchet, Robert Meier, Simon Schug +2
Online learning holds the promise of enabling efficient long-term credit assignment in recurrent neural networks. However, current algorithms fall short of offline backpropagation…
Random initialisations performing above chance and how to find them
Frederik Benzing, Simon Schug, Robert Meier +5
Neural networks trained with stochastic gradient descent (SGD) starting from different random initialisations typically find functionally very similar solutions, raising the questi…
Learning where to learn: Gradient sparsity in meta and continual learning
Johannes von Oswald, Dominic Zhao, Seijin Kobayashi +4
Finding neural network weights that generalize well from small datasets is difficult. A promising approach is to learn a weight initialization such that a small number of weight ch…
A contrastive rule for meta-learning
Nicolas Zucchet, Simon Schug, Johannes von Oswald +2
Humans and other animals are capable of improving their learning performance as they solve related tasks from a given problem domain, to the point of being able to learn from extre…