Scale-invariant Feature Extraction of Neural Network and Renormalization Group Flow

Abstract

Theoretical understanding of how deep neural network (DNN) extracts features from input images is still unclear, but it is widely believed that the extraction is performed hierarchically through a process of coarse-graining. It reminds us of the basic concept of renormalization group (RG) in statistical physics. In order to explore possible relations between DNN and RG, we use the Restricted Boltzmann machine (RBM) applied to Ising model and construct a flow of model parameters (in particular, temperature) generated by the RBM. We show that the unsupervised RBM trained by spin configurations at various temperatures from $T=0$ to $T=6$ generates a flow along which the temperature approaches the critical value $T_c=2.27$. This behavior is opposite to the typical RG flow of the Ising model. By analyzing various properties of the weight matrices of the trained RBM, we discuss why it flows towards $T_c$ and how the RBM learns to extract features of spin configurations.

arXiv comment32 pages, 17 figures

References in corpus (20)

• Discovering Phase Transitions with Unsupervised Learning
• Unsupervised learning of phase transitions: from principal component analysis to variational autoencoders
• Why does deep and cheap learning work so well?
• Discovering Phases, Phase Transitions and Crossovers through Unsupervised Machine Learning: A critical examination
• Learning Thermodynamics with Boltzmann Machines
• Mutual Information, Neural Networks and the Renormalization Group
• Detection of phase transition via convolutional neural network
• An exact mapping between the Variational Renormalization Group and Deep Learning
• Multitasking associative networks
• Deep Learning the Ising Model Near Criticality
• Deep learning and the renormalization group
• Advanced Mean Field Theory of Restricted Boltzmann Machine
• Restricted Boltzmann Machines for the Long Range Ising Models
• Nonequilibrium Thermodynamics of Restricted Boltzmann Machines
• Unsupervised feature learning from finite data by message passing: discontinuous versus continuous phase transition
• Exact Mapping from Singular Value Spectrum of Fractal Images to Entanglement Spectrum of One-Dimensional Quantum Systems
• Why does Deep Learning work? - A perspective from Group Theory
• Entropy of Factorized Snapshot Data for Two-Dimensional Classical Spin Models
• Singular-Value-Decomposition Analysis of Associative Memory in a Neural Network
• Renormalization Group Transformation for Hamiltonian Dynamical Systems in Biological Networks

Cited by in corpus (40)

• Neural Network Renormalization Group
• Machine Learning for Quantum Matter
• Restricted Boltzmann Machines for Quantum States with Nonabelian or Anyonic Symmetries
• Super-resolving the Ising model with convolutional neural networks
• Smallest Neural Network to Learn the Ising Criticality
• Machine Learning as a universal tool for quantitative investigations of phase transition
• Drawing Phase Diagrams of Random Quantum Systems by Deep Learning the Wave Functions
• Thermodynamics and Feature Extraction by Machine Learning
• Learning the Ising Model with Generative Neural Networks
• Machine learning determination of dynamical parameters: The Ising model case
• Interpretable machine learning for inferring the phase boundaries in a nonequilibrium system
• Neural network setups for a precise detection of the many-body localization transition: finite-size scaling and limitations
• Is Deep Learning a Renormalization Group Flow?
• Deep learning black hole metrics from shear viscosity
• Machine Learning Application to Two-Dimensional Dzyaloshinskii-Moriya Ferromagnets
• QuCumber: wavefunction reconstruction with neural networks
• Renormalization Group Flow as Optimal Transport
• Self-learning Monte Carlo method with Behler-Parrinello neural networks
• Phase transition encoded in neural network
• Inverse Renormalization Group based on Image Super-Resolution using Deep Convolutional Networks
• Neural Network flows of low q-state Potts and clock Models
• Neural Monte Carlo Renormalization Group
• Machine learning generated configurations in presence of a conserved quantity: a cautionary tale
• Can a CNN trained on the Ising model detect the phase transition of the $q$-state Potts model?
• Adding machine learning within Hamiltonians: Renormalization group transformations, symmetry breaking and restoration
• Signal detection in nearly continuous spectra and symmetry breaking
• Wavelet Conditional Renormalization Group
• Short sighted deep learning
• Emergence of a finite-size-scaling function in the supervised learning of the Ising phase transition
• Thermodynamics of the Ising model encoded in restricted Boltzmann machines
• Berezinskii-Kosterlitz-Thouless transition from Neural Network Flows
• Logical Reasoning for Revealing the Critical Temperature through Deep Learning of Configuration Ensemble of Statistical Systems
• An Enquiry on similarities between Renormalization Group and Auto-Encoders using Transfer Learning
• Self-regularizing restricted Boltzmann machines
• Randomized-gauge test for machine learning of Ising model order parameter
• Feature extraction of machine learning and phase transition point of Ising model
• Dynamic neuron approach to deep neural networks: Decoupling neurons for renormalization group analysis
• Deep learning of thermodynamic laws from microscopic dynamics
• Maximum Multiscale Entropy and Neural Network Regularization
• Learning disentangled representation for classical models