A random energy approach to deep learning
arXiv:2112.09420 · doi:10.1088/1742-5468/ac7794
Abstract
We study a generic ensemble of deep belief networks which is parametrized by the distribution of energy levels of the hidden states of each layer. We show that, within a random energy approach, statistical dependence can propagate from the visible to deep layers only if each layer is tuned close to the critical point during learning. As a consequence, efficiently trained learning machines are characterised by a broad distribution of energy levels. The analysis of Deep Belief Networks and Restricted Boltzmann Machines on different datasets confirms these conclusions.
arXiv comment16 pages, 4 figures
References in corpus (11)
- Are biological systems poised at criticality?
- Statistical physics of inference: Thresholds and algorithms
- Intrinsic dimension of data representations in deep neural networks
- On sampling and modeling complex systems
- Statistical mechanics of the US Supreme Court
- Statistical Criticality arises in Most Informative Representations
- Learning with hidden variables
- Exploring the Function Space of Deep-Learning Machines
- Minimum Description Length codes are critical
- Space of Functions Computed by Deep-Layered Machines
- The peculiar statistical mechanics of Optimal Learning Machines