Complexity for deep neural networks and other characteristics of deep feature representations
arXiv:2006.04791
Abstract
We define a notion of complexity, which quantifies the nonlinearity of the computation of a neural network, as well as a complementary measure of the effective dimension of feature representations. We investigate these observables both for trained networks for various datasets as well as explore their dynamics during training, uncovering in particular power law scaling. These observables can be understood in a dual way as uncovering hidden internal structure of the datasets themselves as a function of scale or depth. The entropic character of the proposed notion of complexity should allow to transfer modes of analysis from neuroscience and statistical physics to the domain of artificial neural networks. The introduced observables can be applied without any change to the analysis of biological neuronal systems.
arXiv commentSignificant extension including developments in neuroscience context and more. 36 pages
References in corpus (16)
- Weak pairwise correlations imply strongly correlated network states in a neural population
- On the Number of Linear Regions of Deep Neural Networks
- Complexity and Shock Wave Geometries
- Tensorizing Neural Networks
- Advances on Tensor Network Theory: Symmetries, Fermions, Entanglement, and Holography
- Ultimate tensorization: compressing convolutional and FC layers alike
- Supervised Learning with Quantum-Inspired Tensor Networks
- Deep Learning and AdS/CFT
- Path integral optimization as circuit complexity
- AdS/CFT as a deep Boltzmann machine
- Complexity of Linear Regions in Deep Networks
- Deep Learning and Quantum Entanglement: Fundamental Connections with Implications to Network Design
- Analysis and Design of Convolutional Networks via Hierarchical Tensor Decompositions
- Ensemble Inhibition and Excitation in the Human Cortex: an Ising Model Analysis with Uncertainties
- Entropy from Machine Learning
- Spectral Analysis of Latent Representations