Exploring the Function Space of Deep-Learning Machines
arXiv:1708.01422 · doi:10.1103/PhysRevLett.120.248301
Abstract
The function space of deep-learning machines is investigated by studying growth in the entropy of functions of a given error with respect to a reference function, realized by a deep-learning machine. Using physics-inspired methods we study both sparsely and densely-connected architectures to discover a layer-wise convergence of candidate functions, marked by a corresponding reduction in entropy when approaching the reference function, gain insight into the importance of having a large number of layers, and observe phase transitions as the error increases.
arXiv commentNew examples of networks with ReLU activation and convolutional networks are included
References in corpus (9)
- Striving for Simplicity: The All Convolutional Net
- Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
- Understanding Neural Networks Through Deep Visualization
- Deep Neural Networks as Gaussian Processes
- Exploring Generalization in Deep Learning
- Exponential expressivity in deep neural networks through transient chaos
- Parallel dynamics of disordered Ising spin systems on finitely connected random graphs
- Noisy Random Boolean Formulae - a Statistical Physics Perspective
- Computing with Noise - Phase Transitions in Boolean Formulas
Cited by in corpus (14)
- Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
- Drawing Phase Diagrams of Random Quantum Systems by Deep Learning the Wave Functions
- Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach
- Mean-field inference methods for neural networks
- A statistical mechanics framework for Bayesian deep neural networks beyond the infinite-width limit
- Learning Curves for Deep Neural Networks: A Gaussian Field Theory Perspective
- Counting the learnable functions of structured data
- Beyond the storage capacity: data driven satisfiability transition
- Large Deviation Analysis of Function Sensitivity in Random Deep Neural Networks
- Space of Functions Computed by Deep-Layered Machines
- A random energy approach to deep learning
- Initializing ReLU networks in an expressive subspace of weights
- Dynamics of Meta-learning Representation in the Teacher-student Scenario
- No one-hidden-layer neural network can represent multivariable functions