Exploring the Function Space of Deep-Learning Machines
arXiv:1708.01422 · doi:10.1103/PhysRevLett.120.248301
Abstract
The function space of deep-learning machines is investigated by studying growth in the entropy of functions of a given error with respect to a reference function, realized by a deep-learning machine. Using physics-inspired methods we study both sparsely and densely-connected architectures to discover a layer-wise convergence of candidate functions, marked by a corresponding reduction in entropy when approaching the reference function, gain insight into the importance of having a large number of layers, and observe phase transitions as the error increases.
New examples of networks with ReLU activation and convolutional networks are included
References in corpus (4)
Cited by in corpus (14)
- Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
- Drawing Phase Diagrams of Random Quantum Systems by Deep Learning the Wave Functions
- Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach
- Mean-field inference methods for neural networks
- A statistical mechanics framework for Bayesian deep neural networks beyond the infinite-width limit
- Learning Curves for Deep Neural Networks: A Gaussian Field Theory Perspective
- Counting the learnable functions of structured data
- Beyond the storage capacity: data driven satisfiability transition
- Large Deviation Analysis of Function Sensitivity in Random Deep Neural Networks
- Space of Functions Computed by Deep-Layered Machines
- A random energy approach to deep learning
- Initializing ReLU networks in an expressive subspace of weights
- Dynamics of Meta-learning Representation in the Teacher-student Scenario
- No one-hidden-layer neural network can represent multivariable functions