Random active path model of deep neural networks with diluted binary synapses
arXiv:1705.00850 · doi:10.1103/PhysRevE.98.042311
Abstract
Deep learning has become a powerful and popular tool for a variety of machine learning tasks. However, it is challenging to understand the mechanism of deep learning from a theoretical perspective. In this work, we propose a random active path model to study collective properties of deep neural networks with binary synapses, under the removal perturbation of connections between layers. In the model, the path from input to output is randomly activated, and the corresponding input unit constrains the weights along the path into the form of a -weight interaction glass model. A critical value of the perturbation is observed to separate a spin glass regime from a paramagnetic regime, with the transition being of the first order. The paramagnetic phase is conjectured to have a poor generalization performance.
arXiv comment10 pages, 5 figures, with Supplemental Material (upon request)
References in corpus (8)
- Proceedings of the 29th International Conference on Machine Learning (ICML-12)
- The Bethe lattice spin glass revisited
- On the nature of the low-temperature phase in discontinuous mean-field spin glasses
- A ferromagnet with a glass transition
- On the Robustness of Convolutional Neural Networks to Internal Architecture and Weight Perturbations
- On the energy landscape of deep networks
- Theory of population coupling and applications to describe high order correlations in large populations of interacting neurons
- Reinforced stochastic gradient descent for deep neural network learning