Towards a generalization of information theory for hierarchical partitions

Abstract

Complex systems often exhibit multiple levels of organization covering a wide range of physical scales, so the study of the hierarchical decomposition of their structure and function is frequently convenient. To better understand this phenomenon, we introduce a generalization of information theory that works with hierarchical partitions. We begin revisiting the recently introduced Hierarchical Mutual Information (HMI), and show that it can be written as a level by level summation of classical conditional mutual information terms. Then, we prove that the HMI is bounded from above by the corresponding hierarchical joint entropy. In this way, in analogy to the classical case, we derive hierarchical generalizations of many other classical information-theoretic quantities. In particular, we prove that, as opposed to its classical counterpart, the hierarchical generalization of the Variation of Information is not a metric distance, but it admits a transformation into one. Moreover, focusing on potential applications of the existing developments of the theory, we show how to adjust by chance the HMI. We also corroborate and analyze all the presented theoretical results with exhaustive numerical computations, and include an illustrative application example of the introduced formalism. Finally, we mention some open problems that should be eventually addressed for the proposed generalization of information theory to reach maturity.

arXiv comment6 figures

References in corpus (24)

• Comparing community structure identification
• Hierarchical Organization in Complex Networks
• Community detection in networks: A user guide
• Self-similar community structure in organisations
• Self-similarity of complex networks
• The statistical mechanics of networks
• Saving Human Lives: What Complexity Science and Information Systems can Contribute
• Extracting the hierarchical organization of complex systems
• Multilevel compression of random walks on networks reveals hierarchical organization in large integrated systems
• Memory in network flows and its effects on spreading dynamics and community detection
• The Statistical Physics of Real-World Networks
• Hierarchical Block Structures and High-resolution Model Selection in Large Networks
• Multiscale unfolding of real networks by geometric renormalization
• Element-centric clustering comparison unifies overlaps and hierarchy
• A Mini-Introduction To Information Theory
• Evaluating accuracy of community detection using the relative normalized mutual information
• Improved mutual information measure for classification and community detection
• Efficient community detection of network flows for varying Markov times and bipartite networks
• Identifying the structural discontinuities of human interactions
• Hierarchical benchmark graphs for testing community detection algorithms
• Extracting tag hierarchies
• Hierarchical mutual information for the comparison of hierarchical community structures in complex networks
• Hierarchical burst model for complex bursty dynamics
• Comparing the hierarchy of keywords in on-line news portals

Cited by in corpus (1)

• Diffusion geometry of multiplex and interdependent systems