Quantifying unique information

Abstract

We propose new measures of shared information, unique information and synergistic information that can be used to decompose the multi-information of a pair of random variables $(Y,Z)$ with a third random variable $X$. Our measures are motivated by an operational idea of unique information which suggests that shared information and unique information should depend only on the pair marginal distributions of $(X,Y)$ and $(X,Z)$. Although this invariance property has not been studied before, it is satisfied by other proposed measures of shared information. The invariance property does not uniquely determine our new measures, but it implies that the functions that we define are bounds to any other measures satisfying the same invariance property. We study properties of our measures and compare them to other candidate measures.

24 pages, 2 figures. Version 2 contains less typos than version 1

References in corpus (3)

• Nonnegative Decomposition of Multivariate Information
• Reconsidering unique information: Towards a multivariate information decomposition
• The Blackwell relation defines no lattice

Cited by in corpus (83)

• Exploration of synergistic and redundant information sharing in static and dynamical Gaussian systems
• Partial Information Decomposition as a Unified Approach to the Specification of Neural Goal Functions
• Information Flows? A Critique of Transfer Entropies
• Measuring multivariate redundant information with pointwise common change in surprisal
• Efficient transfer entropy analysis of non-stationary neural time series
• Bits from Biology for Computational Intelligence
• Measuring Integrated Information: Comparison of Candidate Measures in Theory and Simulation
• IDTxl: The Information Dynamics Toolkit xl: a Python package for the efficient analysis of multivariate information dynamics in networks
• Multiscale Information Decomposition: Exact Computation for Multivariate Gaussian Processes
• Bits and Pieces: Understanding Information Decomposition from Part-whole Relationships and Formal Logic
• Pointwise Partial Information Decomposition using the Specificity and Ambiguity Lattices
• Multivariate Dependence Beyond Shannon Information
• A Novel Approach to the Partial Information Decomposition
• Invariant components of synergy, redundancy, and unique information among three variables
• Redundancy and synergy in dual decompositions of mutual information gain and information loss
• Introducing a differentiable measure of pointwise shared information
• An operational information decomposition via synergistic disclosure
• Understanding interdependency through complex information sharing
• Topological Information Data Analysis
• Quantifying synergistic information using intermediate stochastic variables
• Quantifying Redundant Information in Predicting a Target Random Variable
• Information flow through a model of the C. elegans klinotaxis circuit
• A comparison of partial information decompositions using data from real and simulated layer 5b pyramidal cells
• Quantifying how much sensory information in a neural code is relevant for behavior
• Secret Sharing and Shared Information
• Decomposing past and future: Integrated information decomposition based on shared probability mass exclusions
• BROJA-2PID: A robust estimator for bivariate partial information decomposition
• Unique Information and Secret Key Agreement
• Evaluating Morphological Computation in Muscle and DC-motor Driven Models of Human Hopping
• Reconsidering unique information: Towards a multivariate information decomposition
• Generalised Measures of Multivariate Information Content
• Synergy as a warning sign of transitions: the case of the two-dimensional Ising model
• On extractable shared information
• Estimating the Mutual Information between two Discrete, Asymmetric Variables with Limited Samples
• Information Flow in Computational Systems
• Computing the Unique Information
• Coarse-graining and the Blackwell order
• A Graph Algorithmic Approach to Separate Direct from Indirect Neural Interactions
• Exact partial information decompositions for Gaussian systems based on dependency constraints
• Biological Information
• The Partial Entropy Decomposition: Decomposing multivariate entropy and mutual information via pointwise common surprisal
• Unique Informations and Deficiencies
• Understanding Convolutional Neural Networks with Information Theory: An Initial Exploration
• An information-theoretic approach to hypergraph psychometrics
• Synergistic information in a dynamical model implemented on the human structural connectome reveals spatially distinct associations with age
• The Blackwell relation defines no lattice
• infotheory: A C++/Python package for multivariate information theoretic analysis
• A Framework for Causal Discovery in non-intervenable systems
• A partial information decomposition for discrete and continuous variables
• A General Framework for Interpretable Neural Learning based on Local Information-Theoretic Goal Functions
• Partial Information Decomposition via Deficiency for Multivariate Gaussians
• The identity of information: how deterministic dependencies constrain information synergy and redundancy
• Interplay of synergy and redundancy in diamond motif
• Synergy, Redundancy and Common Information
• A Rigorous Information-Theoretic Definition of Redundancy and Relevancy in Feature Selection Based on (Partial) Information Decomposition
• Pooling Probability Distributions and the Partial Information Decomposition
• Orders between channels and implications for partial information decomposition
• A Measure of Synergy based on Union Information
• Elements of Consciousness and Cognition. Biology, Mathematic, Physics and Panpsychism: an Information Topology Perspective
• The Variational Deficiency Bottleneck
• MAXENT3D_PID: An Estimator for the Maximum-entropy Trivariate Partial Information Decomposition
• Unique Information and Secret Key Decompositions
• Contrasting information theoretic decompositions of modulatory and arithmetic interactions in neural information processing systems
• Information Decomposition on Structured Space
• Can Information Flows Suggest Targets for Interventions in Neural Circuits?
• Properties of Unique Information
• Quantifying multivariate redundancy with maximum entropy decompositions of mutual information
• Explicit Formula for Partial Information Decomposition
• Optimizing Bivariate Partial Information Decomposition
• Disentanglement Analysis with Partial Information Decomposition
• Multiscale Structure of More-than-Binary Variables
• On Information Links
• Some new insights into information decomposition in complex systems based on common information
• Outsourcing Control requires Control Complexity
• Multivariate Partial Information Decomposition: Constructions, Inconsistencies, and Alternative Measures
• Partial information decomposition: redundancy as information bottleneck
• Intersectional synergies: untangling irreducible effects of intersecting identities via information decomposition
• The many faces of deep learning
• Hierarchical Quantification of Synergy in Channels
• Multiscale Partial Information Decomposition of Dynamic Processes with Short and Long-range correlations: Theory and Application to Cardiovascular Control
• Partial Information Decomposition of Boolean Functions: a Fourier Analysis perspective
• Quantifying Influence and Information Transfer in a Modified Vicsek Model with Non-reciprocal Interactions
• An Approximation Scheme for Multivariate Information based on Partial Information Decomposition