89 citations · 328 across the 17 of their papers we have counts for
14 papers · 1 filter
Uncovering mesa-optimization algorithms in Transformers
Johannes von Oswald, Maximilian Schlegel, Alexander Meulemans +10
Some autoregressive models exhibit in-context learning capabilities: being able to learn as an input sequence is processed, without undergoing any parameter changes, and without be…
Gated recurrent neural networks discover attention
Nicolas Zucchet, Seijin Kobayashi, Yassir Akram +4
Recent architectural developments have enabled recurrent neural networks (RNNs) to reach and even surpass the performance of Transformers on certain sequence modeling tasks. These…
Online learning of long-range dependencies
Nicolas Zucchet, Robert Meier, Simon Schug +2
Online learning holds the promise of enabling efficient long-term credit assignment in recurrent neural networks. However, current algorithms fall short of offline backpropagation…
Transformers learn in-context by gradient descent
Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo +4
At present, the mechanisms of in-context learning in Transformers are not well understood and remain mostly an intuition. In this paper, we suggest that training Transformers on au…
The least-control principle for local learning at equilibrium
Alexander Meulemans, Nicolas Zucchet, Seijin Kobayashi +2
Equilibrium systems are a powerful way to express neural computations. As special cases, they include models of great current interest in both neuroscience and machine learning, su…
Beyond backpropagation: bilevel optimization through implicit differentiation and equilibrium propagation
Nicolas Zucchet, João Sacramento
This paper reviews gradient-based techniques to solve bilevel optimization problems. Bilevel optimization is a general way to frame the learning of systems that are implicitly defi…
Learning where to learn: Gradient sparsity in meta and continual learning
Johannes von Oswald, Dominic Zhao, Seijin Kobayashi +4
Finding neural network weights that generalize well from small datasets is difficult. A promising approach is to learn a weight initialization such that a small number of weight ch…
Credit Assignment in Neural Networks through Deep Feedback Control
Alexander Meulemans, Matilde Tristany Farinha, Javier García Ordóñez +3
The success of deep learning sparked interest in whether the brain learns by using similar techniques for assigning credit to each synaptic weight for its contribution to the netwo…
A contrastive rule for meta-learning
Nicolas Zucchet, Simon Schug, Johannes von Oswald +2
Humans and other animals are capable of improving their learning performance as they solve related tasks from a given problem domain, to the point of being able to learn from extre…
Posterior Meta-Replay for Continual Learning
Christian Henning, Maria R. Cervera, Francesco D'Angelo +6
Learning a sequence of tasks without access to i.i.d. observations is a widely studied form of continual learning (CL) that remains challenging. In principle, Bayesian learning dir…
Neural networks with late-phase weights
Johannes von Oswald, Seijin Kobayashi, Alexander Meulemans +3
The largely successful method of training neural networks is to learn their weights using some variant of stochastic gradient descent (SGD). Here, we show that the solutions found…
A Theoretical Framework for Target Propagation
Alexander Meulemans, Francesco S. Carzaniga, Johan A. K. Suykens +2
The success of deep learning, a brain-inspired form of AI, has sparked interest in understanding how the brain could similarly learn across multiple layers of neurons. However, the…
Continual learning with hypernetworks
Johannes von Oswald, Christian Henning, Benjamin F. Grewe +1
Artificial neural networks suffer from catastrophic forgetting when they are sequentially trained on multiple tasks. To overcome this problem, we present a novel approach based on…
Feedforward Initialization for Fast Inference of Deep Generative Networks is biologically plausible
Yoshua Bengio, Benjamin Scellier, Olexa Bilaniuk +2
We consider deep multi-layered generative models such as Boltzmann machines or Hopfield nets in which computation (which implements inference) is both recurrent and stochastic, but…