Markov Categories and Entropy
arXiv:2212.11719 · doi:10.1109/TIT.2023.3328825
Abstract
Markov categories are a novel framework to describe and treat problems in probability and information theory. In this work we combine the categorical formalism with the traditional quantitative notions of entropy, mutual information, and data processing inequalities. We show that several quantitative aspects of information theory can be captured by an enriched version of Markov categories, where the spaces of morphisms are equipped with a divergence or even a metric. As it is customary in information theory, mutual information can be defined as a measure of how far a joint source is from displaying independence of its components. More strikingly, Markov categories give a notion of determinism for sources and channels, and we can define entropy exactly by measuring how far a source or channel is from being deterministic. This recovers Shannon and Rényi entropies, as well as the Gini-Simpson index used in ecology to quantify diversity, and it can be used to give a conceptual definition of generalized entropy.
arXiv comment54 pages
References in corpus (23)
- Rényi Divergence and Kullback-Leibler Divergence
- A synthetic approach to Markov kernels, conditional independence and theorems on sufficient statistics
- A Characterization of Entropy in Terms of Information Loss
- Linear entropy as an entanglement measure in two-fermion systems
- Disintegration and Bayesian Inversion via String Diagrams
- Representable Markov Categories and Comparison of Statistical Experiments in Categorical Probability
- De Finetti's Theorem in Categorical Probability
- Infinite products and zero-one laws in categorical probability
- A Bayesian Characterization of Relative Entropy
- Free gs-monoidal categories and free Markov categories
- A category-theoretic proof of the ergodic decomposition theorem
- Dilations and information flow axioms in categorical probability
- The d-separation criterion in Categorical Probability
- The information loss of a stochastic map
- A functorial characterization of von Neumann entropy
- Probability monads with submonads of deterministic states - Extended version
- Entropy as a Topological Operad Derivation
- Towards a functorial description of quantum relative entropy
- Polynomial Functors and Shannon Entropy
- Sufficient Statistics and Split Idempotents in Discrete Probability Theory
- A Topos Theoretic Notion of Entropy
- On a 2-relative entropy
- Tropical Limits of Probability Spaces, Part I: The Intrinsic Kolmogorov-Sinai Distance and the Asymptotic Equipartition Property for Configurations