Measuring multivariate redundant information with pointwise common change in surprisal

Abstract

The problem of how to properly quantify redundant information is an open question that has been the subject of much recent research. Redundant information refers to information about a target variable S that is common to two or more predictor variables Xi. It can be thought of as quantifying overlapping information content or similarities in the representation of S between the Xi. We present a new measure of redundancy which measures the common change in surprisal shared between variables at the local or pointwise level. We provide a game-theoretic operational definition of unique information, and use this to derive constraints which are used to obtain a maximum entropy distribution. Redundancy is then calculated from this maximum entropy distribution by counting only those local co-information terms which admit an unambiguous interpretation as redundant information. We show how this redundancy measure can be used within the framework of the Partial Information Decomposition (PID) to give an intuitive decomposition of the multivariate mutual information into redundant, unique and synergistic contributions. We compare our new measure to existing approaches over a range of example systems, including continuous Gaussian variables. Matlab code for the measure is provided, including all considered examples.

v3: revisions based on review process at Entropy (expand game-theory and max-ent motivation), v2: add game-theoretic operational definition for maximum entropy constraints; remove thresholding and normalisation of values on lattice

References in corpus (6)

• Partial Information Decomposition as a Unified Approach to the Specification of Neural Goal Functions
• Measuring multivariate redundant information with pointwise common change in surprisal
• Multivariate Dependence Beyond Shannon Information
• Redundancy and synergy in dual decompositions of mutual information gain and information loss
• Secret Sharing and Shared Information
• Reconsidering unique information: Towards a multivariate information decomposition

Cited by in corpus (36)

• Quantifying High-order Interdependencies via Multivariate Extensions of the Mutual Information
• Reconciling emergences: An information-theoretic approach to identify causal emergence in multivariate data
• Measuring multivariate redundant information with pointwise common change in surprisal
• Measuring Integrated Information: Comparison of Candidate Measures in Theory and Simulation
• Pointwise Partial Information Decomposition using the Specificity and Ambiguity Lattices
• Multivariate Dependence Beyond Shannon Information
• Integrated information as a common signature of dynamical and information-processing complexity
• Decomposing causality into its synergistic, unique, and redundant components
• A Novel Approach to the Partial Information Decomposition
• Invariant components of synergy, redundancy, and unique information among three variables
• Redundancy and synergy in dual decompositions of mutual information gain and information loss
• An information-theoretic approach to self-organisation: Emergence of complex interdependencies in coupled dynamical systems
• Introducing a differentiable measure of pointwise shared information
• An operational information decomposition via synergistic disclosure
• A comparison of partial information decompositions using data from real and simulated layer 5b pyramidal cells
• Secret Sharing and Shared Information
• Decomposing past and future: Integrated information decomposition based on shared probability mass exclusions
• Unique Information and Secret Key Agreement
• Generalised Measures of Multivariate Information Content
• Estimating the Mutual Information between two Discrete, Asymmetric Variables with Limited Samples
• Information Based Centralization of Locomotion in Animals and Robots
• InfoAT: Improving Adversarial Training Using the Information Bottleneck Principle
• Exact partial information decompositions for Gaussian systems based on dependency constraints
• Biological Information
• Thermodynamics of exponential Kolmogorov-Nagumo averages
• A General Framework for Interpretable Neural Learning based on Local Information-Theoretic Goal Functions
• The identity of information: how deterministic dependencies constrain information synergy and redundancy
• Pooling Probability Distributions and the Partial Information Decomposition
• Orders between channels and implications for partial information decomposition
• Thermodynamics of memory erasure via a spin reservoir
• A Measure of Synergy based on Union Information
• MAXENT3D_PID: An Estimator for the Maximum-entropy Trivariate Partial Information Decomposition
• Explicit Formula for Partial Information Decomposition
• Multivariate Partial Information Decomposition: Constructions, Inconsistencies, and Alternative Measures
• Quantifying Influence and Information Transfer in a Modified Vicsek Model with Non-reciprocal Interactions
• Information causality beyond the random access code model