89 citations · 328 across the 17 of their papers we have counts for
21 papers
Uncovering mesa-optimization algorithms in Transformers
Johannes von Oswald, Maximilian Schlegel, Alexander Meulemans +10
Some autoregressive models exhibit in-context learning capabilities: being able to learn as an input sequence is processed, without undergoing any parameter changes, and without be…
Gated recurrent neural networks discover attention
Nicolas Zucchet, Seijin Kobayashi, Yassir Akram +4
Recent architectural developments have enabled recurrent neural networks (RNNs) to reach and even surpass the performance of Transformers on certain sequence modeling tasks. These…
Online learning of long-range dependencies
Nicolas Zucchet, Robert Meier, Simon Schug +2
Online learning holds the promise of enabling efficient long-term credit assignment in recurrent neural networks. However, current algorithms fall short of offline backpropagation…
Transformers learn in-context by gradient descent
Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo +4
At present, the mechanisms of in-context learning in Transformers are not well understood and remain mostly an intuition. In this paper, we suggest that training Transformers on au…
The least-control principle for local learning at equilibrium
Alexander Meulemans, Nicolas Zucchet, Seijin Kobayashi +2
Equilibrium systems are a powerful way to express neural computations. As special cases, they include models of great current interest in both neuroscience and machine learning, su…
Beyond backpropagation: bilevel optimization through implicit differentiation and equilibrium propagation
Nicolas Zucchet, João Sacramento
This paper reviews gradient-based techniques to solve bilevel optimization problems. Bilevel optimization is a general way to frame the learning of systems that are implicitly defi…
Minimizing Control for Credit Assignment with Strong Feedback
Alexander Meulemans, Matilde Tristany Farinha, Maria R. Cervera +2
The success of deep learning ignited interest in whether the brain learns hierarchical representations using gradient-based learning. However, current biologically plausible method…
Learning where to learn: Gradient sparsity in meta and continual learning
Johannes von Oswald, Dominic Zhao, Seijin Kobayashi +4
Finding neural network weights that generalize well from small datasets is difficult. A promising approach is to learn a weight initialization such that a small number of weight ch…
Credit Assignment in Neural Networks through Deep Feedback Control
Alexander Meulemans, Matilde Tristany Farinha, Javier García Ordóñez +3
The success of deep learning sparked interest in whether the brain learns by using similar techniques for assigning credit to each synaptic weight for its contribution to the netwo…
Conductance-based dendrites perform Bayes-optimal cue integration
Jakob Jordan, João Sacramento, Willem A. M. Wybo +2
A fundamental function of cortical circuits is the integration of information from different sources to form a reliable basis for behavior. While animals behave as if they optimall…
A contrastive rule for meta-learning
Nicolas Zucchet, Simon Schug, Johannes von Oswald +2
Humans and other animals are capable of improving their learning performance as they solve related tasks from a given problem domain, to the point of being able to learn from extre…
Posterior Meta-Replay for Continual Learning
Christian Henning, Maria R. Cervera, Francesco D'Angelo +6
Learning a sequence of tasks without access to i.i.d. observations is a widely studied form of continual learning (CL) that remains challenging. In principle, Bayesian learning dir…
Neural networks with late-phase weights
Johannes von Oswald, Seijin Kobayashi, Alexander Meulemans +3
The largely successful method of training neural networks is to learn their weights using some variant of stochastic gradient descent (SGD). Here, we show that the solutions found…
Conductance-based dendrites perform reliability-weighted opinion pooling
Jakob Jordan, João Sacramento, Mihai A. Petrovici +1
Cue integration, the combination of different sources of information to reduce uncertainty, is a fundamental computational principle of brain function. Starting from a normative mo…
A Theoretical Framework for Target Propagation
Alexander Meulemans, Francesco S. Carzaniga, Johan A. K. Suykens +2
The success of deep learning, a brain-inspired form of AI, has sparked interest in understanding how the brain could similarly learn across multiple layers of neurons. However, the…
Ghost Units Yield Biologically Plausible Backprop in Deep Neural Networks
Thomas Mesnard, Gaetan Vignoud, Joao Sacramento +2
In the past few years, deep learning has transformed artificial intelligence research and led to impressive performance in various difficult tasks. However, it is still unclear how…
Continual learning with hypernetworks
Johannes von Oswald, Christian Henning, Benjamin F. Grewe +1
Artificial neural networks suffer from catastrophic forgetting when they are sequentially trained on multiple tasks. To overcome this problem, we present a novel approach based on…
Dendritic cortical microcircuits approximate the backpropagation algorithm
João Sacramento, Rui Ponte Costa, Yoshua Bengio +1
Deep learning has seen remarkable developments over the last years, many of them inspired by neuroscience. However, the main learning mechanism behind these advances - error backpr…
Dendritic error backpropagation in deep cortical microcircuits
João Sacramento, Rui Ponte Costa, Yoshua Bengio +1
Animal behaviour depends on learning to associate sensory stimuli with the desired motor command. Understanding how the brain orchestrates the necessary synaptic modifications acro…
Feedforward Initialization for Fast Inference of Deep Generative Networks is biologically plausible
Yoshua Bengio, Benjamin Scellier, Olexa Bilaniuk +2
We consider deep multi-layered generative models such as Boltzmann machines or Hopfield nets in which computation (which implements inference) is both recurrent and stochastic, but…
Binary Willshaw learning yields high synaptic capacity for long-term familiarity memory
João Sacramento, Andreas Wichert
We investigate from a computational perspective the efficiency of the Willshaw synaptic update rule in the context of familiarity discrimination, a binary-answer, memory-related ta…