An exact mapping between the Variational Renormalization Group and Deep Learning

Abstract

Deep learning is a broad set of techniques that uses multiple layers of representation to automatically learn relevant features directly from structured data. Recently, such techniques have yielded record-breaking results on a diverse set of difficult machine learning tasks in computer vision, speech recognition, and natural language processing. Despite the enormous success of deep learning, relatively little is understood theoretically about why these techniques are so successful at feature learning and compression. Here, we show that deep learning is intimately related to one of the most important and successful techniques in theoretical physics, the renormalization group (RG). RG is an iterative coarse-graining scheme that allows for the extraction of relevant features (i.e. operators) as a physical system is examined at different length scales. We construct an exact mapping from the variational renormalization group, first introduced by Kadanoff, and deep learning architectures based on Restricted Boltzmann Machines (RBMs). We illustrate these ideas using the nearest-neighbor Ising Model in one and two-dimensions. Our results suggests that deep learning algorithms may be employing a generalized RG-like scheme to learn relevant features from data.

8 pages, 3 figures

Cited by in corpus (76)

• Geometric deep learning: going beyond Euclidean data
• Machine learning and the physical sciences
• Colloquium: Criticality and dynamical scaling in living systems
• Quantum Entanglement in Neural Network States
• Machine learning quantum phases of matter beyond the fermion sign problem
• Provable approximation properties for deep neural networks
• Non-perturbative methodologies for low-dimensional strongly-correlated systems: From non-abelian bosonization to truncated spectrum methods
• Supervised Learning with Quantum-Inspired Tensor Networks
• Machine learning active-nematic hydrodynamics
• Scale-invariant Feature Extraction of Neural Network and Renormalization Group Flow
• Deep Learning the Ising Model Near Criticality
• Unifying Neural-network Quantum States and Correlator Product States via Tensor Networks
• A Probabilistic Theory of Deep Learning
• On Interpretability of Artificial Neural Networks: A Survey
• Hints of dark energy anisotropic stress using Machine Learning
• Machine learning \& artificial intelligence in the quantum domain
• Disorder Averaging and its UV (Dis)Contents
• Nonperturbative renormalization for the neural network-QFT correspondence
• Deep Learning and Quantum Entanglement: Fundamental Connections with Implications to Network Design
• Neural networks with redundant representation: detecting the undetectable
• Is Deep Learning a Renormalization Group Flow?
• Enforcing constraints for interpolation and extrapolation in Generative Adversarial Networks
• A Deep Learning Approach to Unsupervised Ensemble Learning
• Opening the black box of deep learning
• Inverse Renormalization Group based on Image Super-Resolution using Deep Convolutional Networks
• Generation of ice states through deep reinforcement learning
• Towards meaningful physics from generative models
• RG-Flow: A hierarchical and explainable flow model based on renormalization group and sparse prior
• Why does Deep Learning work? - A perspective from Group Theory
• Non-Gaussian processes and neural networks at finite widths
• Tensor network language model
• Entanglement Area Law for Shallow and Deep Quantum Neural Network States
• Flow of Information in Feed-Forward Deep Neural Networks
• A mathematical motivation for complex-valued convolutional networks
• Interpreting Deep Learning: The Machine Learning Rorschach Test?
• Machine learning algorithms based on generalized Gibbs ensembles
• Why Unsupervised Deep Networks Generalize
• Short sighted deep learning
• Misanthropic Entropy and Renormalization as a Communication Channel
• Self-Supervised Learning of Generative Spin-Glasses with Normalizing Flows
• Interpolating between boolean and extremely high noisy patterns through Minimal Dense Associative Memories
• A Perspective on Deep Imaging
• A Neural Network Perturbation Theory Based on the Born Series
• Renormalization Group as a Koopman Operator
• Variational approach to unsupervised learning
• Elements of Consciousness and Cognition. Biology, Mathematic, Physics and Panpsychism: an Information Topology Perspective
• Entanglement transitions from restricted Boltzmann machines
• Criticality & Deep Learning II: Momentum Renormalisation Group
• Logical Reasoning for Revealing the Critical Temperature through Deep Learning of Configuration Ensemble of Statistical Systems
• What Really is Deep Learning Doing?
• The Autodidactic Universe
• Between Homomorphic Signal Processing and Deep Neural Networks: Constructing Deep Algorithms for Polyphonic Music Transcription
• Neural Network Statistical Mechanics
• Inversion of lattice models from the observations of microscopic degrees of freedom: parameter estimation with uncertainty quantification
• PAC-Bayesian Generalization Bounds for MultiLayer Perceptrons
• A Probabilistic Representation of Deep Learning for Improving The Information Theoretic Interpretability
• Mean Field Theory of Activation Functions in Deep Neural Networks
• The Physics of Machine Learning: An Intuitive Introduction for the Physical Scientist
• A Probabilistic Representation of Deep Learning
• Measure, Manifold, Learning, and Optimization: A Theory Of Neural Networks
• Bayesian Convolutional Neural Networks for Compressed Sensing Restoration
• Fuzzy Logic Interpretation of Quadratic Networks
• Multilevel Anomaly Detection for Mixed Data
• Feature extraction of machine learning and phase transition point of Ising model
• A quantitative definition of organismality and its application to lichen
• Explicitly Bayesian Regularizations in Deep Learning
• Self-regularizing restricted Boltzmann machines
• Resolution and Relevance Trade-offs in Deep Learning
• Renormalization Group Transformation for Hamiltonian Dynamical Systems in Biological Networks
• Optimal Real-Space Renormalization-Group Transformations with Artificial Neural Networks
• Maximum Multiscale Entropy and Neural Network Regularization
• PCANet: An energy perspective
• Super-resolution of spin configurations based on flow-based generative models
• Efficient algorithms for topological inference on random graphs
• The many faces of deep learning
• Network Horizon Dynamics I: Qualitative Aspects