28 citations · 109 across the 19 of their papers we have counts for
7 papers · 1 filter
Emergent temporal abstractions in autoregressive models enable hierarchical reinforcement learning
Seijin Kobayashi, Yanick Schimpf, Maximilian Schlegel +12
Large-scale autoregressive models pretrained on next-token prediction and finetuned with reinforcement learning (RL) have achieved unprecedented success on many problem domains. Du…
MesaNet: Sequence Modeling by Locally Optimal Test-Time Training
Johannes von Oswald, Nino Scherrer, Seijin Kobayashi +14
Sequence modeling is currently dominated by causal transformer architectures that use softmax self-attention. Although widely adopted, transformers require scaling memory and compu…
Not Every AI Problem is a Data Problem: We Should Be Intentional About Data Scaling
Tanya Rodchenko, Natasha Noy, Nino Scherrer
While Large Language Models require more and more data to train and scale, rather than looking for any data to acquire, we should consider what types of tasks are more likely to be…
Uncovering mesa-optimization algorithms in Transformers
Johannes von Oswald, Maximilian Schlegel, Alexander Meulemans +10
Some autoregressive models exhibit in-context learning capabilities: being able to learn as an input sequence is processed, without undergoing any parameter changes, and without be…
Federated Causal Discovery From Interventions
Amin Abyaneh, Nino Scherrer, Patrick Schwab +3
Causal discovery serves a pivotal role in mitigating model uncertainty through recovering the underlying causal mechanisms among variables. In many practical domains, such as healt…
On the Generalization and Adaption Performance of Causal Models
Nino Scherrer, Anirudh Goyal, Stefan Bauer +2
Learning models that offer robust out-of-distribution generalization and fast adaptation is a key challenge in modern machine learning. Modelling causal structure into neural netwo…
Variational Causal Networks: Approximate Bayesian Inference over Causal Structures
Yashas Annadani, Jonas Rothfuss, Alexandre Lacoste +4
Learning the causal structure that underlies data is a crucial step towards robust real-world decision making. The majority of existing work in causal inference focuses on determin…