Mechanisms of dimensionality reduction and decorrelation in deep neural networks
arXiv:1710.01467 · doi:10.1103/PhysRevE.98.062313
Abstract
Deep neural networks are widely used in various domains. However, the nature of computations at each layer of the deep networks is far from being well understood. Increasing the interpretability of deep neural networks is thus important. Here, we construct a mean-field framework to understand how compact representations are developed across layers, not only in deterministic deep networks with random weights but also in generative deep networks where an unsupervised learning is carried out. Our theory shows that the deep computation implements a dimensionality reduction while maintaining a finite level of weak correlations between neurons for possible feature extraction. Mechanisms of dimensionality reduction and decorrelation are unified in the same framework. This work may pave the way for understanding how a sensory hierarchy works.
arXiv comment11 pages, 5 figures, a physics explanation of decorrelation and dimensionality reduction is added; to be published by Phys Rev E (2018)
References in corpus (12)
- Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Brain's Ventral Visual Pathway
- Emergence of Compositional Representations in Restricted Boltzmann Machines
- Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priors
- Spectral Dynamics of Learning Restricted Boltzmann Machines
- Advanced Mean Field Theory of Restricted Boltzmann Machine
- The role of the interaction matrix in mean-field spin glasses
- Statistical mechanics of unsupervised feature learning in a restricted Boltzmann machine with binary synapses
- Susceptibility Propagation by Using Diagonal Consistency
- Unsupervised feature learning from finite data by message passing: discontinuous versus continuous phase transition
- A mean field method with correlations determined by linear response
- Adaptive Thouless-Anderson-Palmer approach to inverse Ising problems with quenched random fields
- Susceptibility Propagation for Constraint Satisfaction Problems
Cited by in corpus (12)
- Intrinsic dimension of data representations in deep neural networks
- Dimension of activity in random neural networks
- On Interpretability of Artificial Neural Networks: A Survey
- Dense Hebbian neural networks: a replica symmetric picture of supervised learning
- Minimal model of permutation symmetry in unsupervised learning
- Decomposing neural networks as mappings of correlation functions
- Eight challenges in developing theory of intelligence
- Sampling scheme for neuromorphic simulation of entangled quantum systems
- Weakly-correlated synapses promote dimension reduction in deep neural networks
- Relationship between manifold smoothness and adversarial vulnerability in deep learning with local errors
- Channel-Wise Early Stopping without a Validation Set via NNK Polytope Interpolation
- Fermi-Bose Machine achieves both generalization and adversarial robustness