Learn your entropy from informative data: an axiom ensuring the consistent identification of generalized entropies

Abstract

Shannon entropy, a cornerstone of information theory, statistical physics and inference methods, is uniquely identified by the Shannon-Khinchin or Shore-Johnson axioms. Generalizations of Shannon entropy, motivated by the study of non-extensive or non-ergodic systems, relax some of these axioms and lead to entropy families indexed by certain `entropic' parameters. In general, the selection of these parameters requires pre-knowledge of the system or encounters inconsistencies. Here we introduce a simple axiom for any entropy family: namely, that no entropic parameter can be inferred from a completely uninformative (uniform) probability distribution. When applied to the Uffink-Jizba-Korbel and Hanel-Thurner entropies, the axiom selects only Rényi entropy as viable. It also extends consistency with the Maximum Likelihood principle, which can then be generalized to estimate the entropic parameter purely from data, as we confirm numerically. Remarkably, in a generalized maximum-entropy framework the axiom implies that the maximized log-likelihood always equals minus Shannon entropy, even if the inferred probability distribution maximizes a generalized entropy and not Shannon's, solving a series of problems encountered in previous approaches.

References in corpus (8)

• Maximum likelihood: extracting unbiased information from complex networks
• From QCD-based hard-scattering to nonextensive statistical mechanical descriptions of transverse momentum spectra in high-energy $pp$ and $p\bar{p}$ collisions
• Tunneling in a very slow ion-molecule reaction
• Evidence of $q$-exponential statistics in Greek seismicity
• Tsallis cosmology and its applications in dark matter physics with focus on IceCube high-energy neutrino data
• Strong ensemble nonequivalence in systems with local constraints
• Comparative study of the heavy-quark dynamics with the Fokker-Planck Equation and the Plastino-Plastino Equation
• Neural complexity -- Statistical-mechanical approach of human electroencephalograms

Cited by in corpus (1)

• Description length of canonical and microcanonical models