Entropy, Information, and the Updating of Probabilities

Abstract

This paper is a review of a particular approach to the method of maximum entropy as a general framework for inference. The discussion emphasizes the pragmatic elements in the derivation. An epistemic notion of information is defined in terms of its relation to the Bayesian beliefs of ideally rational agents. The method of updating from a prior to a posterior probability distribution is designed through an eliminative induction process. The logarithmic relative entropy is singled out as the unique tool for updating that (a) is of universal applicability; (b) that recognizes the value of prior information; and (c) that recognizes the privileged role played by the notion of independence in science. The resulting framework -- the ME method -- can handle arbitrary priors and arbitrary constraints. It includes MaxEnt and Bayes' rule as special cases and, therefore, it unifies entropic and Bayesian methods into a single general inference scheme. The ME method goes beyond the mere selection of a single posterior, but also addresses the question of how much less probable other distributions might be, which provides a direct bridge to the theories of fluctuations and large deviations.

arXiv comment28 pages. Invited paper to appear in Entropy in the special volume "Statistical Foundations of Entropy", ed. by P. Jizba and J. Korbel. arXiv admin note: text overlap with arXiv:1412.5644

References in corpus (21)

• Superstatistics
• Dynamical foundations of nonextensive statistical mechanics
• Updating Probabilities
• A critique of non-extensive q-entropy for thermal statistics
• Maximum Entropy and Bayesian Data Analysis: Entropic Priors
• Maximum Entropy Principle in statistical inference: case for non-Shannonian entropies
• Beyond Boltzmann-Gibbs statistics: Maximum entropy hyperensembles out-of-equilibrium
• Nonadditive Entropies Yield Probability Distributions with Biases not Warranted by the Data
• Updating Probabilities with Data and Moments
• Relative Entropy and Inductive Inference
• Foundations of Inference
• The Entropic Dynamics approach to Quantum Mechanics
• Entropic Dynamics and the Quantum Measurement Problem
• Quantum Measurement and Weak Values in Entropic Dynamics
• Entropic Updating of Probability and Density Matrices
• Conceptual Inadequacy of the Shore and Johnson Axioms for Wide Classes of Complex Systems
• A maximum entropy model of bounded rational decision-making with prior beliefs and market feedback
• Entropic dynamics on Gibbs statistical manifolds
• A Comment on the Tsallis Maximum Entropy Principle
• Information and Entropy
• The Entropic Dynamics of Relativistic Quantum Fields

Cited by in corpus (5)

• A superstatistical measure of distance from canonical equilibrium
• Non-Equilibrium Statistical Physics Beyond the Ideal Heat Bath Approximation
• A Statistical Physics perspective on fairness in shared expenses: The bar bill analogy
• Generalized Derangetropy Functionals for Modeling Cyclical Information Flow
• Kappa distributions in the framework of superstatistics