Renormalized Mutual Information for Artificial Scientific Discovery
arXiv:2005.01912 · doi:10.1103/PhysRevLett.126.200601
Abstract
We derive a well-defined renormalized version of mutual information that allows to estimate the dependence between continuous random variables in the important case when one is deterministically dependent on the other. This is the situation relevant for feature extraction, where the goal is to produce a low-dimensional effective description of a high-dimensional system. Our approach enables the discovery of collective variables in physical systems, thus adding to the toolbox of artificial scientific discovery, while also aiding the analysis of information flow in artificial neural networks.
arXiv commentAdded a more detailed introduction and link to code repository. Physics-based examples and Feature Extraction section have been updated
References in corpus (14)
- Estimating Mutual Information
- Machine learning and the physical sciences
- The information bottleneck method
- Learning deep representations by mutual information estimation and maximization
- A high-bias, low-variance introduction to Machine Learning for physicists
- Opening the Black Box of Deep Neural Networks via Information
- Discovering physical concepts with neural networks
- On Mutual Information Maximization for Representation Learning
- Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle
- Mutual Information, Neural Networks and the Renormalization Group
- Entropy and mutual information in models of deep neural networks
- Toward an AI Physicist for Unsupervised Learning
- Information Loss in Static Nonlinearities
- On the Information Loss in Memoryless Systems: The Multivariate Case