Introducing a differentiable measure of pointwise shared information

Abstract

Partial information decomposition (PID) of the multivariate mutual information describes the distinct ways in which a set of source variables contains information about a target variable. The groundbreaking work of Williams and Beer has shown that this decomposition cannot be determined from classic information theory without making additional assumptions, and several candidate measures have been proposed, often drawing on principles from related fields such as decision theory. None of these measures is differentiable with respect to the underlying probability mass function. We here present a novel measure that satisfies this property, emerges solely from information-theoretic principles, and has the form of a local mutual information. We show how the measure can be understood from the perspective of exclusions of probability mass, a principle that is foundational to the original definition of the mutual information by Fano. Since our measure is well-defined for individual realizations of the random variables it lends itself for example to local learning in artificial neural networks. We also show that it has a meaningful Möbius inversion on a redundancy lattice and obeys a target chain rule. We give an operational interpretation of the measure based on the decisions that an agent should take if given only the shared information.

arXiv comment19 pages, 6 figures; title modified, text modified, typos corrected, manuscript published

References in corpus (16)

• Measuring Information Transfer
• Local information transfer as a spatiotemporal filter for complex systems
• Quantifying unique information
• Nonnegative Decomposition of Multivariate Information
• A Bivariate Measure of Redundant Information
• Partial Information Decomposition as a Unified Approach to the Specification of Neural Goal Functions
• Measuring multivariate redundant information with pointwise common change in surprisal
• Bits from Biology for Computational Intelligence
• IDTxl: The Information Dynamics Toolkit xl: a Python package for the efficient analysis of multivariate information dynamics in networks
• Shared Information -- New Insights and Problems in Decomposing Information in Complex Systems
• Bits and Pieces: Understanding Information Decomposition from Part-whole Relationships and Formal Logic
• Pointwise Partial Information Decomposition using the Specificity and Ambiguity Lattices
• Quantifying synergistic information using intermediate stochastic variables
• On extractable shared information
• Probability Mass Exclusions and the Directed Components of Pointwise Mutual Information
• Optimizing Bivariate Partial Information Decomposition

Cited by in corpus (9)

• Bits and Pieces: Understanding Information Decomposition from Part-whole Relationships and Formal Logic
• A comparison of partial information decompositions using data from real and simulated layer 5b pyramidal cells
• Decomposing past and future: Integrated information decomposition based on shared probability mass exclusions
• Quantifying high-order interdependencies on individual patterns via the local O-information: theory and applications to music analysis
• A General Framework for Interpretable Neural Learning based on Local Information-Theoretic Goal Functions
• A partial information decomposition for discrete and continuous variables
• A Rigorous Information-Theoretic Definition of Redundancy and Relevancy in Feature Selection Based on (Partial) Information Decomposition
• Intersectional synergies: untangling irreducible effects of intersecting identities via information decomposition
• Partial Information Decomposition of Boolean Functions: a Fourier Analysis perspective