Stochastic Synapses Enable Efficient Brain-Inspired Learning Machines

Abstract

Recent studies have shown that synaptic unreliability is a robust and sufficient mechanism for inducing the stochasticity observed in cortex. Here, we introduce Synaptic Sampling Machines, a class of neural network models that uses synaptic stochasticity as a means to Monte Carlo sampling and unsupervised learning. Similar to the original formulation of Boltzmann machines, these models can be viewed as a stochastic counterpart of Hopfield networks, but where stochasticity is induced by a random mask over the connections. Synaptic stochasticity plays the dual role of an efficient mechanism for sampling, and a regularizer during learning akin to DropConnect. A local synaptic plasticity rule implementing an event-driven form of contrastive divergence enables the learning of generative models in an on-line fashion. Synaptic sampling machines perform equally well using discrete-timed artificial units (as in Hopfield networks) or continuous-timed leaky integrate & fire neurons. The learned representations are remarkably sparse and robust to reductions in bit precision and synapse pruning: removal of more than 75% of the weakest connections followed by cursory re-learning causes a negligible performance loss on benchmark classification tasks. The spiking neuron-based synaptic sampling machines outperform existing spike-based unsupervised learners, while potentially offering substantial advantages in terms of power and complexity, and are thus promising models for on-line learning in brain-inspired hardware.

References in corpus (5)

• Proceedings of the 29th International Conference on Machine Learning (ICML-12)
• Spiking Deep Networks with LIF Neurons
• Rounding Methods for Neural Networks with Low Resolution Synaptic Weights
• Probabilistic inference in discrete spaces can be implemented into networks of LIF neurons
• Synaptic sampling: A connection between PSP variability and uncertainty explains neurophysiological observations

Cited by in corpus (23)

• Spiking Neural Networks Hardware Implementations and Challenges: a Survey
• Neuromorphic Deep Learning Machines
• Adaptive Extreme Edge Computing for Wearable Devices
• A recipe for creating ideal hybrid memristive-CMOS neuromorphic computing systems
• A neuromorphic systems approach to in-memory computing with non-ideal memristive devices: From mitigation to exploitation
• Spiking Deep Residual Network
• Neural Sampling Machine with Stochastic Synapse allows Brain-like Learning and Inference
• Pruning of Deep Spiking Neural Networks through Gradient Rewiring
• Stochasticity from function -- why the Bayesian brain may need no noise
• Deterministic networks for probabilistic computing
• Probabilistic Memristive Networks: Application of a Master Equation to Networks of Binary ReRAM cells
• Stochastic single flux quantum neuromorphic computing using magnetically tunable Josephson junctions
• Analog Signal Processing Using Stochastic Magnets
• ConvPath: A Software Tool for Lung Adenocarcinoma Digital Pathological Image Analysis Aided by Convolutional Neural Network
• Fast and energy-efficient neuromorphic deep learning with first-spike times
• Deep Neuromorphic Networks with Superconducting Single Flux Quanta
• The Discrete Langevin Machine: Bridging the Gap Between Thermodynamic and Neuromorphic Systems
• Binary stochasticity enabled highly efficient neuromorphic deep learning achieves better-than-software accuracy
• Coherent noise enables probabilistic sequence replay in spiking neuronal networks
• A synapse-centric account of the free energy principle
• Inherent Weight Normalization in Stochastic Neural Networks
• Estimation of Energy-dissipation Lower-bounds for Neuromorphic Learning-in-memory
• Locally Learned Synaptic Dropout for Complete Bayesian Inference