A Bayesian Characterization of Relative Entropy
arXiv:1402.3067
Abstract
We give a new characterization of relative entropy, also known as the Kullback-Leibler divergence. We use a number of interesting categories related to probability theory. In particular, we consider a category FinStat where an object is a finite set equipped with a probability distribution, while a morphism is a measure-preserving function together with a stochastic right inverse . The function can be thought of as a measurement process, while s provides a hypothesis about the state of the measured system given the result of a measurement. Given this data we can define the entropy of the probability distribution on relative to the "prior" given by pushing the probability distribution on forwards along . We say that is "optimal" if these distributions agree. We show that any convex linear, lower semicontinuous functor from FinStat to the additive monoid which vanishes when is optimal must be a scalar multiple of this relative entropy. Our proof is independent of all earlier characterizations, but inspired by the work of Petz.
arXiv comment32 pages, minor revision
References in corpus (2)
Cited by in corpus (12)
- A mathematical theory of resources
- Pairwise maximum-entropy models and their Glauber dynamics: bimodality, bistability, non-ergodicity problems, and their elimination via inhibition
- Gamma Spaces and Information
- Markov Categories and Entropy
- Landauer's principle as a special case of Galois connection
- Bayesian inversion and the Tomita-Takesaki modular group
- Non-commutative disintegrations: existence and uniqueness in finite dimensions
- Stinespring's construction as an adjunction
- Polynomial Functors and Shannon Entropy
- On a 2-relative entropy
- On Information Links
- A categorical characterization of relative entropy on standard Borel spaces