Understanding interdependency through complex information sharing

Abstract

The interactions between three or more random variables are often nontrivial, poorly understood, and yet, are paramount for future advances in fields such as network information theory, neuroscience, genetics and many others. In this work, we propose to analyze these interactions as different modes of information sharing. Towards this end, we introduce a novel axiomatic framework for decomposing the joint entropy, which characterizes the various ways in which random variables can share information. The key contribution of our framework is to distinguish between interdependencies where the information is shared redundantly, and synergistic interdependencies where the sharing structure exists in the whole but not between the parts. We show that our axioms determine unique formulas for all the terms of the proposed decomposition for a number of cases of interest. Moreover, we show how these results can be applied to several network information theory problems, providing a more intuitive understanding of their fundamental limits.

39 pages, 4 figures

References in corpus (3)

• Nonnegative Decomposition of Multivariate Information
• Exploration of synergistic and redundant information sharing in static and dynamical Gaussian systems
• On the sufficiency of pairwise interactions in maximum entropy models of biological networks

Cited by in corpus (16)

• Quantifying High-order Interdependencies via Multivariate Extensions of the Mutual Information
• Measuring Integrated Information: Comparison of Candidate Measures in Theory and Simulation
• Pointwise Partial Information Decomposition using the Specificity and Ambiguity Lattices
• Multivariate Dependence Beyond Shannon Information
• Invariant components of synergy, redundancy, and unique information among three variables
• Generative power of a protein language model trained on multiple sequence alignments
• An information-theoretic approach to self-organisation: Emergence of complex interdependencies in coupled dynamical systems
• An operational information decomposition via synergistic disclosure
• Unique Information and Secret Key Agreement
• Generalised Measures of Multivariate Information Content
• Quantifying high-order interdependencies on individual patterns via the local O-information: theory and applications to music analysis
• A generalization of the maximum entropy principle for curved statistical manifolds
• An information-theoretic approach to hypergraph psychometrics
• Quantifying multivariate redundancy with maximum entropy decompositions of mutual information
• Understanding complexity via network theory: a gentle introduction
• Partial information decomposition: redundancy as information bottleneck