The three faces of entropy for complex systems -- information, thermodynamics and the maxent principle
arXiv:1705.07714 · doi:10.1103/PhysRevE.96.032124
Abstract
There are three ways to conceptualize entropy: entropy as an extensive thermodynamic quantity of physical systems (Clausius, Boltzmann, Gibbs), entropy as a measure for information production of ergodic sources (Shannon), and entropy as a means for statistical inference on multinomial Bernoulli processes (Jaynes maximum entropy principle). Even though these notions are fundamentally different concepts, the functional form of the entropy for thermodynamic systems in equilibrium, for ergodic sources in information theory, and for independent sampling processes in statistical systems, is degenerate, . For many complex systems, which are typically history-dependent, non-ergodic and non-multinomial, this is no longer the case. Here we show that for such processes the three entropy concepts lead to different functional forms of entropy. We explicitly compute these entropy functionals for three concrete examples. For Polya urn processes, which are simple self-reinforcing processes, the source information rate is , the thermodynamical (extensive) entropy is -entropy, , and the entropy in the maxent principle (MEP) is . For sample space reducing (SSR) processes, which are simple path-dependent processes that are associated with power law statistics, the information rate is , the extensive entropy is , and the maxent result is . Finally, for multinomial mixture processes, the information rate is given by the conditional entropy , with respect to the mixing kernel , the extensive entropy is given by , and the MEP functional corresponds one-to-one to the logarithm of the mixing kernel.
arXiv comment12 pages, 2 figures
References in corpus (7)
- When do generalized entropies apply? How phase space volume determines entropy
- Understanding scaling through history-dependent processes with collapsing sample space
- How multiplicity determines entropy and the derivation of the maximum entropy principle for complex systems
- Generalized (c,d)-entropy and aging random walks
- Sample space reducing cascading processes produce the full spectrum of scaling exponents
- Extreme robustness of scaling in sample space reducing processes explains Zipf's law in diffusion on directed networks
- Calculation of the transition matrix and of the occupation probabilities for the states of the Oslo sandpile model
Cited by in corpus (13)
- Classification of complex systems by their sample-space scaling exponents
- When Shannon and Khinchin meet Shore and Johnson: equivalence of information theory and statistical inference axiomatics
- Temperature is not an observable in superstatistics
- Coupled VAE: Improved Accuracy and Robustness of a Variational Autoencoder
- Information geometric duality of -deformed exponential families
- Information geometry of scaling expansions of non-exponentially growing configuration spaces
- Decomposing information into copying versus transformation
- Maximum configuration principle for driven systems with arbitrary driving
- Relating complexities for the reflexive study of complex systems
- A Mutation Threshold for Cooperative Takeover
- Learn your entropy from informative data: an axiom ensuring the consistent identification of generalized entropies
- Reduced state of the field and classicality of quantum Gaussian evolution
- Gintropy: Gini index based generalization of Entropy