Space of Functions Computed by Deep-Layered Machines
arXiv:2004.08930 · doi:10.1103/PhysRevLett.125.168301
Abstract
We study the space of functions computed by random-layered machines, including deep neural networks and Boolean circuits. Investigating the distribution of Boolean functions computed on the recurrent and layer-dependent architectures, we find that it is the same in both models. Depending on the initial conditions and computing elements used, we characterize the space of functions computed at the large depth limit and show that the macroscopic entropy of Boolean functions is either monotonically increasing or decreasing with the growing depth.
References in corpus (18)
- Adam: A Method for Stochastic Optimization
- Deep Neural Networks as Gaussian Processes
- Exponential expressivity in deep neural networks through transient chaos
- Multitasking associative networks
- Mean-field message-passing equations in the Hopfield model and its generalizations
- Path Integral Approach to Random Neural Networks
- Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priors
- Parallel dynamics of disordered Ising spin systems on finitely connected random graphs
- A Fine-Grained Spectral Perspective on Neural Networks
- Advanced Mean Field Theory of Restricted Boltzmann Machine
- Statistical learning theory of structured data
- Noisy Random Boolean Formulae - a Statistical Physics Perspective
- Exploring the Function Space of Deep-Learning Machines
- Beyond the storage capacity: data driven satisfiability transition
- Computing with Noise - Phase Transitions in Boolean Formulas
- The Boolean Functions Computed by Random Boolean Formulas OR How to Grow the Right Function
- Large Deviation Analysis of Function Sensitivity in Random Deep Neural Networks
- Structure of attractors in randomly connected networks
Cited by in corpus (5)
- A statistical mechanics framework for Bayesian deep neural networks beyond the infinite-width limit
- Unified field theoretical approach to deep and recurrent neuronal networks
- Activation function dependence of the storage capacity of treelike neural networks
- A random energy approach to deep learning
- Dynamics of Meta-learning Representation in the Teacher-student Scenario