Is Deep Learning a Renormalization Group Flow?

Abstract

Although there has been a rapid development of practical applications, theoretical explanations of deep learning are in their infancy. Deep learning performs a sophisticated coarse graining. Since coarse graining is a key ingredient of the renormalization group (RG), RG may provide a useful theoretical framework directly relevant to deep learning. In this study we pursue this possibility. A statistical mechanics model for a magnet, the Ising model, is used to train an unsupervised restricted Boltzmann machine (RBM). The patterns generated by the trained RBM are compared to the configurations generated through an RG treatment of the Ising model. Although we are motivated by the connection between deep learning and RG flow, in this study we focus mainly on comparing a single layer of a deep network to a single step in the RG flow. We argue that correlation functions between hidden and visible neurons are capable of diagnosing RG-like coarse graining. Numerical experiments show the presence of RG-like patterns in correlators computed using the trained RBMs. The observables we consider are also able to exhibit important differences between RG and deep learning.

arXiv commentThis article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2020.3000901, IEEE Access

References in corpus (19)

• Machine learning phases of matter
• Entanglement renormalization
• Understanding deep learning requires rethinking generalization
• The Conformal Bootstrap: Theory, Numerical Techniques, and Applications
• Low-Rank Tensor Networks for Dimensionality Reduction and Large-Scale Optimization Problems: Perspectives and Challenges PART 1
• Why does deep and cheap learning work so well?
• Tensor Network Renormalization
• Tensor Networks for Dimensionality Reduction and Large-Scale Optimizations. Part 2 Applications and Future Perspectives
• Mutual Information, Neural Networks and the Renormalization Group
• An exact mapping between the Variational Renormalization Group and Deep Learning
• Forest fires and other examples of self-organized criticality
• Real Space Renormalization in Statistical Mechanics
• Scale-invariant Feature Extraction of Neural Network and Renormalization Group Flow
• Deep Learning the Ising Model Near Criticality
• Renormalization group approach to the critical behavior of the forest fire model
• Dynamic Renormalization Group Approach to Self-Organized Critical Phenomena
• Thermodynamics and Feature Extraction by Machine Learning
• Deep learning and the renormalization group
• Why does Deep Learning work? - A perspective from Group Theory

Cited by in corpus (23)

• The Calabi-Yau Landscape: from Geometry, to Physics, to Machine-Learning
• Nonperturbative renormalization for the neural network-QFT correspondence
• Renormalization Group Flow as Optimal Transport
• Machine Learning Lie Structures & Applications to Physics
• Neural Network flows of low q-state Potts and clock Models
• Field theoretical approach for signal detection in nearly continuous positive spectra II: Tensorial data
• A Triumvirate of AI Driven Theoretical Discovery
• Signal detection in nearly continuous spectra and symmetry breaking
• Machine learning for structure-property relationships: Scalability and limitations
• Why Unsupervised Deep Networks Generalize
• Short sighted deep learning
• Universes as Big Data
• Thermodynamics of the Ising model encoded in restricted Boltzmann machines
• Some Open Questions in Quiver Gauge Theory
• Berezinskii-Kosterlitz-Thouless transition from Neural Network Flows
• Black holes and the loss landscape in machine learning
• Exploring explicit coarse-grained structure in artificial neural networks
• The Autodidactic Universe
• Functional renormalization group for signal detection and stochastic ergodicity breaking
• Dynamic neuron approach to deep neural networks: Decoupling neurons for renormalization group analysis
• A Two-Phase Perspective on Deep Learning Dynamics
• Dreaming up scale invariance via inverse renormalization group
• Functional Renormalization Group Approach for Signal Detection