A Characterization of Entropy in Terms of Information Loss
arXiv:1106.1791 · doi:10.3390/e13111945
Abstract
There are numerous characterizations of Shannon entropy and Tsallis entropy as measures of information obeying certain properties. Using work by Faddeev and Furuichi, we derive a very simple characterization. Instead of focusing on the entropy of a probability measure on a finite set, this characterization focuses on the `information loss', or change in entropy, associated with a measure-preserving function. Information loss is a special case of conditional entropy: namely, it is the entropy of a random variable conditioned on some function of that variable. We show that Shannon entropy gives the only concept of information loss that is functorial, convex-linear and continuous. This characterization naturally generalizes to Tsallis entropy as well.
11 pages LaTeX, minor revision
References in corpus (1)
Cited by in corpus (45)
- A mathematical theory of resources
- A generalization of majorization that characterizes Shannon entropy
- A non-commutative Bayes' theorem
- Charged Rényi Entropies in CFTs with Einstein-Gauss-Bonnet Holographic Duals
- A Bayesian Characterization of Relative Entropy
- An Inverse Mass Expansion for Entanglement Entropy in Free Massive Scalar Field Theory
- Entropy algebras and Birkhoff factorization
- Gamma Spaces and Information
- Gravity from thermodynamics: optimal transport and negative effective dimensions
- The information loss of a stochastic map
- Markov Categories and Entropy
- Mathematical Characterization of Private and Public Immune Repertoire Sequences
- Point Information Gain and Multidimensional Data Analysis
- A functorial characterization of von Neumann entropy
- What is Entropy? A new perspective from games of chance
- Homotopy Theoretic and Categorical Models of Neural Information Networks
- Entropy and Diversity: The Axiomatic Approach
- Entropy as a Topological Operad Derivation
- Topos and Stacks of Deep Neural Networks
- Elements of Consciousness and Cognition. Biology, Mathematic, Physics and Panpsychism: an Information Topology Perspective
- Towards a functorial description of quantum relative entropy
- Additive Entropies of Partitions
- Polynomial Functors and Shannon Entropy
- Supervised Learning of Labeled Pointcloud Differences via Cover-Tree Entropy Reduction
- Phase space distributions in information theory
- Extending Resource Monotones using Kan Extensions
- The categorical basis of dynamical entropy
- The Census and the Second Law: An Entropic Approach to Optimal Apportionment for the U.S. House of Representatives
- On Information Links
- On a 2-relative entropy
- Quantum conditional entropy from information-theoretic principles
- Tropical Limits of Probability Spaces, Part I: The Intrinsic Kolmogorov-Sinai Distance and the Asymptotic Equipartition Property for Configurations
- Information Measures for Deterministic Input-Output Systems
- Homological Tools for the Quantum Mechanic
- Coherence and strictification for self-similarity
- A categorical characterization of relative entropy on standard Borel spaces
- Sampling and statistical physics via symmetry
- Maximum Entropy competes with Maximum Likelihood
- On Information (pseudo) Metric
- Partially observable systems and quotient entropy via graphs
- Measurements and Information in Spin Foam Models
- Dirichlet polynomials and entropy
- Categorical magnitude and entropy
- The Poincaré-Boltzmann Machine: from Statistical Physics to Machine Learning and back
- A note on Shannon entropy