Power Law in Sparsified Deep Neural Networks
arXiv:1805.01891
Abstract
The power law has been observed in the degree distributions of many biological neural networks. Sparse deep neural networks, which learn an economical representation from the data, resemble biological neural networks in many ways. In this paper, we study if these artificial networks also exhibit properties of the power law. Experimental results on two popular deep learning models, namely, multilayer perceptrons and convolutional neural networks, are affirmative. The power law is also naturally related to preferential attachment. To study the dynamical properties of deep networks in continual learning, we propose an internal preferential attachment model to explain how the network topology evolves. Experimental results show that with the arrival of a new task, the new connections made follow this preferential attachment process.
References in corpus (8)
- Emergence of scaling in random networks
- Overcoming catastrophic forgetting in neural networks
- Classes of behavior of small-world networks
- Scale-free brain functional networks
- Continual Learning Through Synaptic Intelligence
- An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
- PathNet: Evolution Channels Gradient Descent in Super Neural Networks
- Fitting Power-laws in empirical data with estimators that work for all exponents