On the correspondence between thermodynamics and inference

Abstract

We expand upon a natural analogy between Bayesian statistics and statistical physics in which sample size corresponds to inverse temperature. This analogy motivates the definition of two novel statistical quantities: a learning capacity and a Gibbs entropy. The analysis of the learning capacity, corresponding to the heat capacity in thermal physics, leads to new insight into the mechanism of learning and explains why some models have anomalously-high learning performance. We explore the properties of the learning capacity in a number of examples, including a sloppy model. Next, we propose that the Gibbs entropy provides a natural device for counting distinguishable distributions in the context of Bayesian inference. We use this device to define a generalized principle of indifference (GPI) in which every distinguishable model is assigned equal a priori probability. This principle results in a new solution to a long-standing problem in Bayesian inference: the definition of an objective or uninformative prior. A key characteristic of this new approach is that it can be applied to analyses where the model dimension is unknown and circumvents the automatic rejection of higher-dimensional models in Bayesian inference.

13 pages, 6 figures, 2 tables, and appendix

Cited by in corpus (6)

• Marginal likelihood computation for model selection and hypothesis testing: an extensive review
• Manifold learning for parameter reduction
• Bayesian, frequentist, and information geometric approaches to parametric uncertainty quantification of classical empirical interatomic potentials
• A scaling law from discrete to continuous solutions of channel capacity problems in the low-noise limit
• Optimal encoding in stochastic latent-variable Models
• Intrinsic regularization effect in Bayesian nonlinear regression scaled by observed data