Role of zero synapses in unsupervised feature learning

Abstract

Synapses in real neural circuits can take discrete values, including zero (silent or potential) synapses. The computational role of zero synapses in unsupervised feature learning of unlabeled noisy data is still unclear, thus it is important to understand how the sparseness of synaptic activity is shaped during learning and its relationship with receptive field formation. Here, we formulate this kind of sparse feature learning by a statistical mechanics approach. We find that learning decreases the fraction of zero synapses, and when the fraction decreases rapidly around a critical data size, an intrinsically structured receptive field starts to develop. Further increasing the data size refines the receptive field, while a very small fraction of zero synapses remain to act as contour detectors. This phenomenon is discovered not only in learning a handwritten digits dataset, but also in learning retinal neural activity measured in a natural-movie-stimuli experiment.

6 pages, 4 figures, to appear in J. Phys A as a LETTER

References in corpus (6)

• Emergence of Compositional Representations in Restricted Boltzmann Machines
• Extensive load in multitasking associative networks
• Advanced Mean Field Theory of Restricted Boltzmann Machine
• Sparsely-Connected Neural Networks: Towards Efficient VLSI Implementation of Deep Neural Networks
• Unsupervised feature learning from finite data by message passing: discontinuous versus continuous phase transition
• Clustering of neural codewords revealed by a first-order phase transition

Cited by in corpus (7)

• Restricted Boltzmann Machine, recent advances and mean-field theory
• Edge of chaos and avalanches in neural networks with heavy-tailed synaptic weight distribution
• Non-Convex Multi-species Hopfield models
• Modeling Structured Data Learning with Restricted Boltzmann Machines in the Teacher-Student Setting
• Variational mean-field theory for training restricted Boltzmann machines with binary synapses
• Meta predictive learning model of languages in neural circuits
• Saddle Hierarchy in Dense Associative Memory