Publications (19)
A deep learning theory for neural networks grounded in physics
Benjamin Scellier
In the last decade, deep learning has become a major component of artificial intelligence. The workhorse of deep learning is the optimization of loss functions by stochastic gradie…
Training a Predictive Coding Network on ImageNet using Equilibrium Propagation
Tugdual Kerjan, Rasmus Høier, Benjamin Scellier
Equilibrium Propagation (EP) is a physics-based training framework that has primarily been employed in energy-based models, including continuous Hopfield networks, nonlinear resist…
Training of Physical Neural Networks
Ali Momeni, Babak Rahmani, Benjamin Scellier +25
Physical neural networks (PNNs) are a class of neural-like networks that leverage the properties of physical systems to perform computation. While PNNs are so far a niche research…
Equilibrium Propagation with Continual Weight Updates
Maxence Ernoult, Julie Grollier, Damien Querlioz +2
Equilibrium Propagation (EP) is a learning algorithm that bridges Machine Learning and Neuroscience, by computing gradients closely matching those of Backpropagation Through Time (…
Quantum Equilibrium Propagation: Gradient-Descent Training of Quantum Systems
Benjamin Scellier
Equilibrium propagation (EP) is a training framework for energy-based systems, i.e. systems whose physics minimizes an energy function. EP has been explored in various classical ph…
Continual Weight Updates and Convolutional Architectures for Equilibrium Propagation
Maxence Ernoult, Julie Grollier, Damien Querlioz +2
Equilibrium Propagation (EP) is a biologically inspired alternative algorithm to backpropagation (BP) for training neural networks. It applies to RNNs fed by a static input x that…
Agnostic Physics-Driven Deep Learning
Benjamin Scellier, Siddhartha Mishra, Yoshua Bengio +1
This work establishes that a physical system can perform statistical learning without gradient computations, via an Agnostic Equilibrium Propagation (Aeqprop) procedure that combin…
Scaling Equilibrium Propagation to Deep ConvNets by Drastically Reducing its Gradient Estimator Bias
Axel Laborieux, Maxence Ernoult, Benjamin Scellier +3
Equilibrium Propagation (EP) is a biologically-inspired counterpart of Backpropagation Through Time (BPTT) which, owing to its strong theoretical guarantees and the locality in spa…
Scaling Equilibrium Propagation to Deep ConvNets by Drastically Reducing its Gradient Estimator Bias
Axel Laborieux, Maxence Ernoult, Benjamin Scellier +3
Equilibrium Propagation (EP) is a biologically-inspired algorithm for convergent RNNs with a local learning rule that comes with strong theoretical guarantees. The parameter update…
A universal approximation theorem for nonlinear resistive networks
Benjamin Scellier, Siddhartha Mishra
Resistor networks have recently been studied as analog computing platforms for machine learning, particularly due to their compatibility with the Equilibrium Propagation training f…
Energy-based learning algorithms for analog computing: a comparative study
Benjamin Scellier, Maxence Ernoult, Jack Kendall +1
Energy-based learning algorithms have recently gained a surge of interest due to their compatibility with analog (post-digital) hardware. Existing algorithms include contrastive le…
Equivalence of Equilibrium Propagation and Recurrent Backpropagation
Benjamin Scellier, Yoshua Bengio
Recurrent Backpropagation and Equilibrium Propagation are supervised learning algorithms for fixed point recurrent neural networks which differ in their second phase. In the first…
A Fast Algorithm to Simulate Nonlinear Resistive Networks
Benjamin Scellier
Analog electrical networks have long been investigated as energy-efficient computing platforms for machine learning, leveraging analog physics during inference. More recently, resi…
Generalization of Equilibrium Propagation to Vector Field Dynamics
Benjamin Scellier, Anirudh Goyal, Jonathan Binas +2
The biological plausibility of the backpropagation algorithm has long been doubted by neuroscientists. Two major reasons are that neurons would need to send two different types of…
Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation
Benjamin Scellier, Yoshua Bengio
We introduce Equilibrium Propagation, a learning framework for energy-based models. It involves only one kind of neural computation, performed in both the first phase (when the pre…
Feedforward Initialization for Fast Inference of Deep Generative Networks is biologically plausible
Yoshua Bengio, Benjamin Scellier, Olexa Bilaniuk +2
We consider deep multi-layered generative models such as Boltzmann machines or Hopfield nets in which computation (which implements inference) is both recurrent and stochastic, but…
Temporal Contrastive Learning through implicit non-equilibrium memory
Martin J. Falk, Adam T. Strupp, Benjamin Scellier +1
The backpropagation method has enabled transformative uses of neural networks. Alternatively, for energy-based models, local learning methods involving only nearby neurons offer be…
Updates of Equilibrium Prop Match Gradients of Backprop Through Time in an RNN with Static Input
Maxence Ernoult, Julie Grollier, Damien Querlioz +2
Equilibrium Propagation (EP) is a biologically inspired learning algorithm for convergent recurrent neural networks, i.e. RNNs that are fed by a static input x and settle to a stea…
Training End-to-End Analog Neural Networks with Equilibrium Propagation
Jack Kendall, Ross Pantone, Kalpana Manickavasagam +2
We introduce a principled method to train end-to-end analog neural networks by stochastic gradient descent. In these analog neural networks, the weights to be adjusted are implemen…