Quantifying Relevance in Learning and Inference

Abstract

Learning is a distinctive feature of intelligent behaviour. High-throughput experimental data and Big Data promise to open new windows on complex systems such as cells, the brain or our societies. Yet, the puzzling success of Artificial Intelligence and Machine Learning shows that we still have a poor conceptual understanding of learning. These applications push statistical inference into uncharted territories where data is high-dimensional and scarce, and prior information on "true" models is scant if not totally absent. Here we review recent progress on understanding learning, based on the notion of "relevance". The relevance, as we define it here, quantifies the amount of information that a dataset or the internal representation of a learning machine contains on the generative model of the data. This allows us to define maximally informative samples, on one hand, and optimal learning machines on the other. These are ideal limits of samples and of machines, that contain the maximal amount of information about the unknown generative process, at a given resolution (or level of compression). Both ideal limits exhibit critical features in the statistical sense: Maximally informative samples are characterised by a power-law frequency distribution (statistical criticality) and optimal learning machines by an anomalously large susceptibility. The trade-off between resolution (i.e. compression) and relevance distinguishes the regime of noisy representations from that of lossy compression. These are separated by a special point characterised by Zipf's law statistics. This identifies samples obeying Zipf's law as the most compressed loss-less representations that are optimal in the sense of maximal relevance. Criticality in optimal learning machines manifests in an exponential degeneracy of energy levels, that leads to unusual thermodynamic properties.

arXiv commentreview article, 63 pages, 14 figures

References in corpus (46)

• GW170817: Observation of Gravitational Waves from a Binary Neutron Star Inspiral
• Power-law distributions in empirical data
• Power laws, Pareto distributions and Zipf's law
• The information bottleneck method
• Direct-coupling analysis of residue co-evolution captures native contacts across many protein families
• Are biological systems poised at criticality?
• Improved contact prediction in proteins: Using pseudolikelihoods to infer Potts models
• Colloquium: Criticality and dynamical scaling in living systems
• Statistical physics of inference: Thresholds and algorithms
• Maximum entropy models for antibody diversity
• Inverse statistical problems: from the inverse Ising problem to data science
• The generalization error of random features regression: Precise asymptotics and double descent curve
• Information-based fitness and the emergence of criticality in living systems
• Multiplicative processes and power laws
• Inferring interaction partners from protein sequences
• Intrinsic dimension of data representations in deep neural networks
• Unreasonable Effectiveness of Learning Neural Networks: From Accessible States and Robust Ensembles to Basic Algorithmic Schemes
• Statistical physics of pairwise probability models
• Emergence of Compositional Representations in Restricted Boltzmann Machines
• Consistency under sampling of exponential random graph models
• Zipf's law and criticality in multivariate data without fine-tuning
• Mean-field message-passing equations in the Hopfield model and its generalizations
• Entropy estimates of small data sets
• On sampling and modeling complex systems
• On the criticality of inferred models
• Data clustering and noise undressing for correlation matrices
• Statistical mechanics of the US Supreme Court
• Determination of the edge of criticality in echo state networks through Fisher information maximization
• Stochastic Thermodynamics of Learning
• Thermodynamics of Restricted Boltzmann Machines and related learning dynamics
• Inference of entropies of discrete random variables with unknown cardinalities
• Statistical Criticality arises in Most Informative Representations
• Learning and inference in a nonequilibrium Ising model with hidden nodes
• Criticality of mostly informative samples: A Bayesian model selection approach
• Learning with hidden variables
• Identifying relevant positions in proteins by Critical Variable Selection
• Counting the learnable functions of structured data
• Minimum Description Length codes are critical
• Fraudulent White Noise: Flat power spectra belie arbitrarily complex processes
• Optimal Work Extraction and the Minimum Description Length Principle
• A random energy approach to deep learning
• Optimal encoding in stochastic latent-variable Models
• Bayesian Inference of Minimally Complex Models with Interactions of Arbitrary Order
• Separation and Concentration in Deep Networks
• The peculiar statistical mechanics of Optimal Learning Machines
• Thermodynamic Machine Learning through Maximum Work Production

Cited by in corpus (9)

• Making sense of complex systems through resolution, relevance, and mapping entropy
• Evaluating Large Language Models in Code Generation: INFINITE Methodology for Defining the Inference Index
• Perceptual Rationality: An Evolutionary Game Theory of Perceptually Rational Decision-Making
• Extrinsic vs Intrinsic Criticality in Systems with Many Components
• On the class of coding optimality of human languages and the origins of Zipf's law
• A New Spin on Color Quantization
• Universal emergence of local Zipf-Mandelbrot law
• Universal Scale Laws for Colors and Patterns in Imagery
• Multiscale Relevance of Natural Images