Generalization learning in a perceptron with binary synapses

Abstract

We consider the generalization problem for a perceptron with binary synapses, implementing the Stochastic Belief-Propagation-Inspired (SBPI) learning algorithm which we proposed earlier, and perform a mean-field calculation to obtain a differential equation which describes the behaviour of the device in the limit of a large number of synapses N. We show that the solving time of SBPI is of order N*sqrt(log(N)), while the similar, well-known clipped perceptron (CP) algorithm does not converge to a solution at all in the time frame we considered. The analysis gives some insight into the ongoing process and shows that, in this context, the SBPI algorithm is equivalent to a new, simpler algorithm, which only differs from the CP algorithm by the addition of a stochastic, unsupervised meta-plastic reinforcement process, whose rate of application must be less than sqrt(2/(π* N)) for the learning to be achieved effectively. The analytical results are confirmed by simulations.

16 pages, 4 figures

References in corpus (3)

• Learning by message-passing in networks of discrete synapses
• Efficient supervised learning in networks with binary synapses
• Stability and noise in biochemical switches

Cited by in corpus (15)

• Unreasonable Effectiveness of Learning Neural Networks: From Accessible States and Robust Ensembles to Basic Algorithmic Schemes
• Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapses
• Shaping the learning landscape in neural networks around wide flat minima
• Efficiency of quantum versus classical annealing in non-convex learning problems
• Local entropy as a measure for sampling solutions in Constraint Satisfaction Problems
• Learning through atypical "phase transitions" in overparameterized neural networks
• Learning may need only a few bits of synaptic precision
• On the role of synaptic stochasticity in training low-precision neural networks
• A Max-Sum algorithm for training discrete neural networks
• Clustering of solutions in the symmetric binary perceptron
• Learning by random walks in the weight space of the Ising perceptron
• Theory and learning protocols for the material tempotron model
• Active online learning in the binary perceptron problem
• Frozen $1$-RSB structure of the symmetric Ising perceptron
• Training Multi-Layer Binary Neural Networks With Local Binary Error Signals