1 citations · 1 across the 10 of their papers we have counts for
12 papers
Retrieval-Centric Deep Learning in Growing Nonparametric Neural Networks
Maximilian Schlegel, Rajai Nasser, Seijin Kobayashi +6
We investigate a general-purpose layer for deep learning that, instead of compressing arbitrary-size training data into fixed-size weight matrices, stores a new pair of key-value r…
A game theory for foundation models shows new paths to rational cooperation through similarity inference
Alexander Meulemans, Maciej Wołczyk, Marissa A. Weis +11
As autonomous agents powered by foundation models are increasingly integrated into social and economic systems, understanding the principles governing their collective behavior is…
Can In-Context Learning Support Intrinsic Curiosity?
Eric Elmoznino, Sangnie Bhardwaj, Johannes von Oswald +5
Effective machine learning depends not only on how we model data, but also on what data we choose to collect. While large sequence models have revolutionized data modeling, the pro…
Multi-agent cooperation through in-context co-player inference
Marissa A. Weis, Maciej Wołczyk, Rajai Nasser +4
Achieving cooperation among self-interested agents remains a fundamental challenge in multi-agent reinforcement learning. Recent work showed that mutual cooperation can be induced…
Emergent temporal abstractions in autoregressive models enable hierarchical reinforcement learning
Seijin Kobayashi, Yanick Schimpf, Maximilian Schlegel +12
Large-scale autoregressive models pretrained on next-token prediction and finetuned with reinforcement learning (RL) have achieved unprecedented success on many problem domains. Du…
Embedded Universal Predictive Intelligence: a coherent framework for multi-agent learning
Alexander Meulemans, Rajai Nasser, Maciej Wołczyk +10
The standard theory of model-free reinforcement learning assumes that the environment dynamics are stationary and that agents are decoupled from their environment, such that polici…
MesaNet: Sequence Modeling by Locally Optimal Test-Time Training
Johannes von Oswald, Nino Scherrer, Seijin Kobayashi +14
Sequence modeling is currently dominated by causal transformer architectures that use softmax self-attention. Although widely adopted, transformers require scaling memory and compu…
Multi-agent cooperation through learning-aware policy gradients
Alexander Meulemans, Seijin Kobayashi, Johannes von Oswald +6
Self-interested individuals often fail to cooperate, posing a fundamental challenge for multi-agent learning. How can we achieve cooperation among self-interested, independent lear…
When can transformers compositionally generalize in-context?
Seijin Kobayashi, Simon Schug, Yassir Akram +5
Many tasks can be composed from a few independent components. This gives rise to a combinatorial explosion of possible tasks, only some of which might be encountered during trainin…
State Soup: In-Context Skill Learning, Retrieval and Mixing
Maciej Pióro, Maciej Wołczyk, Razvan Pascanu +2
A new breed of gated-linear recurrent neural networks has reached state-of-the-art performance on a range of sequence modeling problems. Such models naturally handle long sequences…
Attention as a Hypernetwork
Simon Schug, Seijin Kobayashi, Yassir Akram +2
Transformers can under some circumstances generalize to novel problem instances whose constituent parts might have been encountered during training, but whose compositions have not…
Discovering modular solutions that generalize compositionally
Simon Schug, Seijin Kobayashi, Yassir Akram +6
Many complex tasks can be decomposed into simpler, independent parts. Discovering such underlying compositional structure has the potential to enable compositional generalization.…