Weakly-correlated synapses promote dimension reduction in deep neural networks

Abstract

By controlling synaptic and neural correlations, deep learning has achieved empirical successes in improving classification performances. How synaptic correlations affect neural correlations to produce disentangled hidden representations remains elusive. Here we propose a simplified model of dimension reduction, taking into account pairwise correlations among synapses, to reveal the mechanism underlying how the synaptic correlations affect dimension reduction. Our theory determines the synaptic-correlation scaling form requiring only mathematical self-consistency, for both binary and continuous synapses. The theory also predicts that weakly-correlated synapses encourage dimension reduction compared to their orthogonal counterparts. In addition, these synapses slow down the decorrelation process along the network depth. These two computational roles are explained by the proposed mean-field equation. The theoretical predictions are in excellent agreement with numerical simulations, and the key features are also captured by a deep learning with Hebbian rules.

arXiv comment21 pages, 8 figures

References in corpus (7)

• Reducing Overfitting in Deep Networks by Decorrelating Representations
• Dimensionality compression and expansion in Deep Neural Networks
• Mechanisms of dimensionality reduction and decorrelation in deep neural networks
• Minimal model of permutation symmetry in unsupervised learning
• Statistical physics of unsupervised learning with prior knowledge in neural networks
• A correlation game for unsupervised learning yields computational interpretations of Hebbian excitation, anti-Hebbian inhibition, and synapse elimination
• If deep learning is the answer, then what is the question?

Cited by in corpus (6)

• Decomposing neural networks as mappings of correlation functions
• Eight challenges in developing theory of intelligence
• Relationship between manifold smoothness and adversarial vulnerability in deep learning with local errors
• Eigenvalue spectrum of neural networks with arbitrary Hebbian length
• Network reconstruction may not mean dynamics prediction
• Fermi-Bose Machine achieves both generalization and adversarial robustness