How multiplicity determines entropy and the derivation of the maximum entropy principle for complex systems

Abstract

The maximum entropy principle (MEP) is a method for obtaining the most likely distribution functions of observables from statistical systems, by maximizing entropy under constraints. The MEP has found hundreds of applications in ergodic and Markovian systems in statistical mechanics, information theory, and statistics. For several decades there exists an ongoing controversy whether the notion of the maximum entropy principle can be extended in a meaningful way to non-extensive, non-ergodic, and complex statistical systems and processes. In this paper we start by reviewing how Boltzmann-Gibbs-Shannon entropy is related to multiplicities of independent random processes. We then show how the relaxation of independence naturally leads to the most general entropies that are compatible with the first three Shannon-Khinchin axioms, the (c,d)-entropies. We demonstrate that the MEP is a perfectly consistent concept for non-ergodic and complex statistical systems if their relative entropy can be factored into a generalized multiplicity and a constraint term. The problem of finding such a factorization reduces to finding an appropriate representation of relative entropy in a linear basis. In a particular example we show that path-dependent random processes with memory naturally require specific generalized entropies. The example is the first exact derivation of a generalized entropy from the microscopic properties of a path-dependent random process.

6 pages, 1 figure. To appear in PNAS

References in corpus (3)

• Generalized entropies and logarithms and their duality relations
• Exponential Families and MaxEnt Calculations for Entropy Measures of Statistical Physics
• Entropies for complex systems: generalized-generalized entropies

Cited by in corpus (25)

• Understanding scaling through history-dependent processes with collapsing sample space
• Reconstruction methods for networks: the case of economic and financial systems
• Maximum Entropy Principle in statistical inference: case for non-Shannonian entropies
• Fitting Power-laws in empirical data with estimators that work for all exponents
• Economic resilience from input-output susceptibility improves predictions of economic growth and recovery
• The three faces of entropy for complex systems -- information, thermodynamics and the maxent principle
• Sample space reducing cascading processes produce the full spectrum of scaling exponents
• The 7% Rule: A Maximum Entropy Prediction on New Decays of the Higgs Boson
• Extreme robustness of scaling in sample space reducing processes explains Zipf's law in diffusion on directed networks
• Boltzmann-Gibbs entropy is sufficient but not necessary for the likelihood factorization required by Einstein
• A new entropy based on a group-theoretical structure
• Inter-occurrence times and universal laws in finance, earthquakes and genomes
• Towards a Universal Measure of Complexity
• Open Problems within Nonextensive Statistical Mechanics
• On Statistical Properties of Jizba-Arimitsu Hybrid Entropy
• Incidence of $q$-statistics in rank distributions
• Entropy? Honest!
• Self-Organization of Self-Clearing Beating Patterns in an Array of Locally Interacting Ciliated Cells Formulated as an Adaptive Boolean Network
• From Rényi Entropy Power to Information Scan of Quantum States
• Thermodynamics of structure-forming systems
• Maximum configuration principle for driven systems with arbitrary driving
• Probabilistic Models with Nonlocal Correlations: Numerical Evidence of q-Large Deviation Theory
• Dimensional measures of generalized entropy
• Stripping syntax from complexity: An information-theoretical perspective on complex systems
• Matrix approach to generalized ensemble theory for nonequilibrium discrete systems