5 citations · 5 across the 3 of their papers we have counts for
4 papers · 1 filter
Retrieval-Centric Deep Learning in Growing Nonparametric Neural Networks
Maximilian Schlegel, Rajai Nasser, Seijin Kobayashi +6
We investigate a general-purpose layer for deep learning that, instead of compressing arbitrary-size training data into fixed-size weight matrices, stores a new pair of key-value r…
Emergent temporal abstractions in autoregressive models enable hierarchical reinforcement learning
Seijin Kobayashi, Yanick Schimpf, Maximilian Schlegel +12
Large-scale autoregressive models pretrained on next-token prediction and finetuned with reinforcement learning (RL) have achieved unprecedented success on many problem domains. Du…
MesaNet: Sequence Modeling by Locally Optimal Test-Time Training
Johannes von Oswald, Nino Scherrer, Seijin Kobayashi +14
Sequence modeling is currently dominated by causal transformer architectures that use softmax self-attention. Although widely adopted, transformers require scaling memory and compu…
Uncovering mesa-optimization algorithms in Transformers
Johannes von Oswald, Maximilian Schlegel, Alexander Meulemans +10
Some autoregressive models exhibit in-context learning capabilities: being able to learn as an input sequence is processed, without undergoing any parameter changes, and without be…