Generalised Measures of Multivariate Information Content

Abstract

The entropy of a pair of random variables is commonly depicted using a Venn diagram. This representation is potentially misleading, however, since the multivariate mutual information can be negative. This paper presents new measures of multivariate information content that can be accurately depicted using Venn diagrams for any number of random variables. These measures complement the existing measures of multivariate mutual information and are constructed by considering the algebraic structure of information sharing. It is shown that the distinct ways in which a set of marginal observers can share their information with a non-observing third party corresponds to the elements of a free distributive lattice. The redundancy lattice from partial information decomposition is then subsequently and independently derived by combining the algebraic structures of joint and shared information content.

arXiv comment31 pages, 11 figures

References in corpus (27)

• Quantifying unique information
• Exploration of synergistic and redundant information sharing in static and dynamical Gaussian systems
• Quantifying High-order Interdependencies via Multivariate Extensions of the Mutual Information
• A Bivariate Measure of Redundant Information
• Partial Information Decomposition as a Unified Approach to the Specification of Neural Goal Functions
• Measuring multivariate redundant information with pointwise common change in surprisal
• Large-scale directed network inference with multivariate transfer entropy and hierarchical statistical testing
• IDTxl: The Information Dynamics Toolkit xl: a Python package for the efficient analysis of multivariate information dynamics in networks
• Synergy and redundancy in the Granger causal analysis of dynamical networks
• Multiscale Information Decomposition: Exact Computation for Multivariate Gaussian Processes
• Intersection Information based on Common Randomness
• Expanding the Transfer Entropy to Identify Information Subgraphs in Complex Systems
• Pointwise Partial Information Decomposition using the Specificity and Ambiguity Lattices
• Multivariate Dependence Beyond Shannon Information
• Invariant components of synergy, redundancy, and unique information among three variables
• An information-theoretic approach to self-organisation: Emergence of complex interdependencies in coupled dynamical systems
• Towards a Synergy-based Approach to Measuring Information Modification
• Understanding interdependency through complex information sharing
• Quantifying synergistic information using intermediate stochastic variables
• Quantifying Redundant Information in Predicting a Target Random Variable
• Secret Sharing and Shared Information
• Unique Information and Secret Key Agreement
• Reconsidering unique information: Towards a multivariate information decomposition
• On extractable shared information
• Redundancy in the information transmission in a two-step cascade
• Coarse-graining and the Blackwell order
• Probability Mass Exclusions and the Directed Components of Pointwise Mutual Information

Cited by in corpus (3)

• Decomposing past and future: Integrated information decomposition based on shared probability mass exclusions
• Quantifying high-order interdependencies on individual patterns via the local O-information: theory and applications to music analysis
• Multivariate Partial Information Decomposition: Constructions, Inconsistencies, and Alternative Measures