89 citations · 328 across the 17 of their papers we have counts for
4 papers · 1 filter
Transformers learn in-context by gradient descent
Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo +4
At present, the mechanisms of in-context learning in Transformers are not well understood and remain mostly an intuition. In this paper, we suggest that training Transformers on au…
The least-control principle for local learning at equilibrium
Alexander Meulemans, Nicolas Zucchet, Seijin Kobayashi +2
Equilibrium systems are a powerful way to express neural computations. As special cases, they include models of great current interest in both neuroscience and machine learning, su…
Beyond backpropagation: bilevel optimization through implicit differentiation and equilibrium propagation
Nicolas Zucchet, João Sacramento
This paper reviews gradient-based techniques to solve bilevel optimization problems. Bilevel optimization is a general way to frame the learning of systems that are implicitly defi…
Minimizing Control for Credit Assignment with Strong Feedback
Alexander Meulemans, Matilde Tristany Farinha, Maria R. Cervera +2
The success of deep learning ignited interest in whether the brain learns hierarchical representations using gradient-based learning. However, current biologically plausible method…