89 citations · 163 across the 15 of their papers we have counts for
6 papers · 1 filter
Reasoning with Neural Cellular Automata
Mayalen Etcheverry, Pietro Miotti, Aidan Sirbu +7
Modern AI architectures used to solve visual reasoning tasks typically rely heavily on global connectivity and synchronization. As biological systems demonstrate, though, sophistic…
Diffusion as a Training Curriculum for Timestep-Free Iterative Reasoning
Mariia Drozdova, Aidan Sirbu, Pietro Miotti +4
Diffusion models and recursive reasoners are both iterative, but they carry information across iterations differently. We add a persistent hidden state to a diffusion denoiser and…
Uncovering mesa-optimization algorithms in Transformers
Johannes von Oswald, Maximilian Schlegel, Alexander Meulemans +10
Some autoregressive models exhibit in-context learning capabilities: being able to learn as an input sequence is processed, without undergoing any parameter changes, and without be…
Transformers learn in-context by gradient descent
Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo +4
At present, the mechanisms of in-context learning in Transformers are not well understood and remain mostly an intuition. In this paper, we suggest that training Transformers on au…
NCA: Texture Generation with Ultra-Compact Neural Cellular Automata
Alexander Mordvintsev, Eyvind Niklasson
We study the problem of example-based procedural texture synthesis using highly compact models. Given a sample image, we use differentiable programming to train a generative proces…
MPLP: Learning a Message Passing Learning Protocol
Ettore Randazzo, Eyvind Niklasson, Alexander Mordvintsev
We present a novel method for learning the weights of an artificial neural network - a Message Passing Learning Protocol (MPLP). In MPLP, we abstract every operations occurring in…