Learn your entropy from informative data: an axiom ensuring the consistent identification of generalized entropies

Abstract

Shannon entropy, a cornerstone of information theory, statistical physics and inference methods, is uniquely identified by the Shannon-Khinchin or Shore-Johnson axioms. Generalizations of Shannon entropy, motivated by the study of non-extensive or non-ergodic systems, relax some of these axioms and lead to entropy families indexed by certain `entropic' parameters. In general, the selection of these parameters requires pre-knowledge of the system or encounters inconsistencies. Here we introduce a simple axiom for any entropy family: namely, that no entropic parameter can be inferred from a completely uninformative (uniform) probability distribution. When applied to the Uffink-Jizba-Korbel and Hanel-Thurner entropies, the axiom selects only Rényi entropy as viable. It also extends consistency with the Maximum Likelihood principle, which can then be generalized to estimate the entropic parameter purely from data, as we confirm numerically. Remarkably, in a generalized maximum-entropy framework the axiom implies that the maximized log-likelihood always equals minus Shannon entropy, even if the inferred probability distribution maximizes a generalized entropy and not Shannon's, solving a series of problems encountered in previous approaches.

References in corpus (16)

• The Statistical Physics of Real-World Networks
• Maximum likelihood: extracting unbiased information from complex networks
• Equilibrium Distribution of Heavy Quarks in Fokker-Planck Dynamics
• From QCD-based hard-scattering to nonextensive statistical mechanical descriptions of transverse momentum spectra in high-energy $pp$ and $p\bar{p}$ collisions
• Maximum Entropy Principle in statistical inference: case for non-Shannonian entropies
• Fractals, non-extensive statistics and QCD
• Tunneling in a very slow ion-molecule reaction
• Maximum Renyi entropy principle for systems with power--law Hamiltonian
• Evidence of $q$-exponential statistics in Greek seismicity
• The three faces of entropy for complex systems -- information, thermodynamics and the maxent principle
• Tsallis cosmology and its applications in dark matter physics with focus on IceCube high-energy neutrino data
• Classification of complex systems by their sample-space scaling exponents
• When Shannon and Khinchin meet Shore and Johnson: equivalence of information theory and statistical inference axiomatics
• Strong ensemble nonequivalence in systems with local constraints
• Comparative study of the heavy-quark dynamics with the Fokker-Planck Equation and the Plastino-Plastino Equation
• Neural complexity -- Statistical-mechanical approach of human electroencephalograms

Cited by in corpus (1)

• Description length of canonical and microcanonical models