A Bivariate Measure of Redundant Information

Abstract

We define a measure of redundant information based on projections in the space of probability distributions. Redundant information between random variables is information that is shared between those variables. But in contrast to mutual information, redundant information denotes information that is shared about the outcome of a third variable. Formalizing this concept, and being able to measure it, is required for the non-negative decomposition of mutual information into redundant and synergistic information. Previous attempts to formalize redundant or synergistic information struggle to capture some desired properties. We introduce a new formalism for redundant information and prove that it satisfies all the properties necessary outlined in earlier work, as well as an additional criterion that we propose to be necessary to capture redundancy. We also demonstrate the behaviour of this new measure for several examples, compare it to previous measures and apply it to the decomposition of transfer entropy.

16 pages, 15 figures, 1 table, added citation to Griffith et al 2012, Maurer et al 1999

References in corpus (5)

• ARACNE: An Algorithm for the Reconstruction of Gene Regulatory Networks in a Mammalian Cellular Context
• Nonnegative Decomposition of Multivariate Information
• Generalized Measures of Information Transfer
• Complexity Measures from Interaction Structures
• Quantifying synergistic information remains an unsolved problem

Cited by in corpus (69)

• JIDT: An information-theoretic toolkit for studying the dynamics of complex systems
• Quantifying unique information
• Exploration of synergistic and redundant information sharing in static and dynamical Gaussian systems
• Partial Information Decomposition as a Unified Approach to the Specification of Neural Goal Functions
• Efficient transfer entropy analysis of non-stationary neural time series
• Bits from Biology for Computational Intelligence
• Synergy and redundancy in the Granger causal analysis of dynamical networks
• Multiscale Information Decomposition: Exact Computation for Multivariate Gaussian Processes
• Intersection Information based on Common Randomness
• Shared Information -- New Insights and Problems in Decomposing Information in Complex Systems
• Pointwise Partial Information Decomposition using the Specificity and Ambiguity Lattices
• Multivariate Dependence Beyond Shannon Information
• A Novel Approach to the Partial Information Decomposition
• Invariant components of synergy, redundancy, and unique information among three variables
• Redundancy and synergy in dual decompositions of mutual information gain and information loss
• Introducing a differentiable measure of pointwise shared information
• Understanding interdependency through complex information sharing
• An operational information decomposition via synergistic disclosure
• Synergetic and redundant information flow detected by unnormalized Granger causality: application to resting state fMRI
• Quantifying Redundant Information in Predicting a Target Random Variable
• A comparison of partial information decompositions using data from real and simulated layer 5b pyramidal cells
• Information flow through a model of the C. elegans klinotaxis circuit
• Quantifying how much sensory information in a neural code is relevant for behavior
• Secret Sharing and Shared Information
• BROJA-2PID: A robust estimator for bivariate partial information decomposition
• Decomposing past and future: Integrated information decomposition based on shared probability mass exclusions
• Unique Information and Secret Key Agreement
• Generalised Measures of Multivariate Information Content
• Reconsidering unique information: Towards a multivariate information decomposition
• Quantifying high-order interdependencies on individual patterns via the local O-information: theory and applications to music analysis
• Synergy as a warning sign of transitions: the case of the two-dimensional Ising model
• On extractable shared information
• Estimating the Mutual Information between two Discrete, Asymmetric Variables with Limited Samples
• Information Flow in Computational Systems
• A Graph Algorithmic Approach to Separate Direct from Indirect Neural Interactions
• Exact partial information decompositions for Gaussian systems based on dependency constraints
• Probability Mass Exclusions and the Directed Components of Pointwise Mutual Information
• Biological Information
• The Partial Entropy Decomposition: Decomposing multivariate entropy and mutual information via pointwise common surprisal
• Unique Informations and Deficiencies
• Synergistic information in a dynamical model implemented on the human structural connectome reveals spatially distinct associations with age
• A Framework for Causal Discovery in non-intervenable systems
• A partial information decomposition for discrete and continuous variables
• A General Framework for Interpretable Neural Learning based on Local Information-Theoretic Goal Functions
• Partial Information Decomposition via Deficiency for Multivariate Gaussians
• Interplay of synergy and redundancy in diamond motif
• The identity of information: how deterministic dependencies constrain information synergy and redundancy
• Synergy, Redundancy and Common Information
• Pooling Probability Distributions and the Partial Information Decomposition
• A Rigorous Information-Theoretic Definition of Redundancy and Relevancy in Feature Selection Based on (Partial) Information Decomposition
• Orders between channels and implications for partial information decomposition
• A Measure of Synergy based on Union Information
• The Variational Deficiency Bottleneck
• Contrasting information theoretic decompositions of modulatory and arithmetic interactions in neural information processing systems
• MAXENT3D_PID: An Estimator for the Maximum-entropy Trivariate Partial Information Decomposition
• Explicit Formula for Partial Information Decomposition
• Quantifying multivariate redundancy with maximum entropy decompositions of mutual information
• On shared and multiple information
• Specialists Outperform Generalists in Ensemble Classification
• Properties of Unique Information
• Some new insights into information decomposition in complex systems based on common information
• Multivariate Partial Information Decomposition: Constructions, Inconsistencies, and Alternative Measures
• An Approximation Scheme for Multivariate Information based on Partial Information Decomposition
• Hierarchical Quantification of Synergy in Channels
• Partial Information Decomposition of Boolean Functions: a Fourier Analysis perspective
• Information causality beyond the random access code model
• Multiscale Partial Information Decomposition of Dynamic Processes with Short and Long-range correlations: Theory and Application to Cardiovascular Control
• Quantifying Influence and Information Transfer in a Modified Vicsek Model with Non-reciprocal Interactions
• Irreducibility is Minimum Synergy Among Parts