Maximal Relevance and Optimal Learning Machines

Abstract

We show that the mutual information between the representation of a learning machine and the hidden features that it extracts from data is bounded from below by the relevance, which is the entropy of the model's energy distribution. Models with maximal relevance -- that we call Optimal Learning Machines (OLM) -- are hence expected to extract maximally informative representations. We explore this principle in a range of models. For fully connected Ising models and we show that {\em i)} OLM are characterised by inhomogeneous distributions of couplings, and that {\em ii)} their learning performance is affected by sub-extensive features that are elusive to a thermodynamic treatment. On specific learning tasks, we find that likelihood maximisation is achieved by models with maximal relevance. Training of Restricted Boltzmann Machines on the MNIST benchmark shows that learning is associated with a broadening of the spectrum of energy levels and that the internal representation of the hidden layer approaches the maximal relevance that can be achieved in a finite dataset. Finally, we discuss a Gaussian learning machine that clarifies that learning hidden features is conceptually different from parameter estimation.

28 pages, 9 figures

References in corpus (13)

• Are biological systems poised at criticality?
• Colloquium: Criticality and dynamical scaling in living systems
• Intrinsic dimension of data representations in deep neural networks
• Unreasonable Effectiveness of Learning Neural Networks: From Accessible States and Robust Ensembles to Basic Algorithmic Schemes
• Emergence of Compositional Representations in Restricted Boltzmann Machines
• Zipf's law and criticality in multivariate data without fine-tuning
• Factors determining nestedness in complex networks
• Mean-field message-passing equations in the Hopfield model and its generalizations
• Thermodynamics of Restricted Boltzmann Machines and related learning dynamics
• Statistical Criticality arises in Most Informative Representations
• Criticality of mostly informative samples: A Bayesian model selection approach
• Counting the learnable functions of structured data
• Minimum Description Length codes are critical