Early Inference in Energy-Based Models Approximates Back-Propagation
arXiv:1510.02777
Abstract
We show that Langevin MCMC inference in an energy-based model with latent variables has the property that the early steps of inference, starting from a stationary point, correspond to propagating error gradients into internal layers, similarly to back-propagation. The error that is back-propagated is with respect to visible units that have received an outside driving force pushing them away from the stationary point. Back-propagated error gradients correspond to temporal derivatives of the activation of hidden units. This observation could be an element of a theory for explaining how brains perform credit assignment in deep hierarchies as efficiently as back-propagation does. In this theory, the continuous-valued latent variables correspond to averaged voltage potential (across time, spikes, and possibly neurons in the same minicolumn), and neural computation corresponds to approximate inference and error back-propagation at the same time.
arXiv admin note: text overlap with arXiv:1509.05936
References in corpus (4)
Cited by in corpus (8)
- Demonstration of Decentralized, Physics-Driven Learning
- Towards an integration of deep learning and neuroscience
- STDP as presynaptic activity times rate of change of postsynaptic activity
- Predictive Coding: a Theoretical and Experimental Review
- Feedforward Initialization for Fast Inference of Deep Generative Networks is biologically plausible
- Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation
- Activation Relaxation: A Local Dynamical Approximation to Backpropagation in the Brain
- Shallow Unorganized Neural Networks using Smart Neuron Model for Visual Perception