Statistical Criticality arises in Most Informative Representations

Abstract

We show that statistical criticality, i.e. the occurrence of power law frequency distributions, arises in samples that are maximally informative about the underlying generating process. In order to reach this conclusion, we first identify the frequency with which different outcomes occur in a sample, as the variable carrying useful information on the generative process. The entropy of the frequency, that we call relevance, provides an upper bound to the number of informative bits. This differs from the entropy of the data, that we take as a measure of resolution. Samples that maximise relevance at a given resolution - that we call maximally informative samples - exhibit statistical criticality. In particular, Zipf's law arises at the optimal trade-off between resolution (i.e. compression) and relevance. As a byproduct, we derive a bound of the maximal number of parameters that can be estimated from a dataset, in the absence of prior knowledge on the generative model. Furthermore, we relate criticality to the statistical properties of the representation of the data generating process. We show that, as a consequence of the concentration property of the Asymptotic Equipartition Property, representations that are maximally informative about the data generating process are characterised by an exponential distribution of energy levels. This arises from a principle of minimal entropy, that is conjugate of the maximum entropy principle in statistical mechanics. This explains why statistical criticality requires no parameter fine tuning in maximally informative samples.

21 pages, 3 figures; Corrected Appendix A; Updated journal reference

References in corpus (15)

• The information bottleneck method
• Are biological systems poised at criticality?
• Opening the Black Box of Deep Neural Networks via Information
• Colloquium: Criticality and dynamical scaling in living systems
• Maximum entropy models for antibody diversity
• Information-based fitness and the emergence of criticality in living systems
• Unreasonable Effectiveness of Learning Neural Networks: From Accessible States and Robust Ensembles to Basic Algorithmic Schemes
• Zipf's law and criticality in multivariate data without fine-tuning
• Entropy estimates of small data sets
• On sampling and modeling complex systems
• On the criticality of inferred models
• Statistical mechanics of the US Supreme Court
• Criticality of mostly informative samples: A Bayesian model selection approach
• Identifying relevant positions in proteins by Critical Variable Selection
• Minimum Description Length codes are critical

Cited by in corpus (19)

• Dynamical approach to Zipf's law
• Taylor's law in innovation processes
• Quantifying Relevance in Learning and Inference
• Maximal Diversity and Zipf's Law
• Minimum Description Length codes are critical
• Making sense of complex systems through resolution, relevance, and mapping entropy
• Maximum Entropy approach to multivariate time series randomization
• Optimal Work Extraction and the Minimum Description Length Principle
• A random energy approach to deep learning
• Optimal encoding in stochastic latent-variable Models
• Two halves of a meaningful text are statistically different
• Damping effect in innovation processes: case studies from Twitter
• The peculiar statistical mechanics of Optimal Learning Machines
• Intelligence, physics and information -- the tradeoff between accuracy and simplicity in machine learning
• Extrinsic vs Intrinsic Criticality in Systems with Many Components
• Scale-invariant representation of machine learning
• Filtering Statistics on Networks
• Maximal Relevance and Optimal Learning Machines
• Complex distributions emerging in filtering and compression