Shared Information -- New Insights and Problems in Decomposing Information in Complex Systems

Abstract

How can the information that a set $X_1,...,X_n$ of random variables contains about another random variable $S$ be decomposed? To what extent do different subgroups provide the same, i.e. shared or redundant, information, carry unique information or interact for the emergence of synergistic information? Recently Williams and Beer proposed such a decomposition based on natural properties for shared information. While these properties fix the structure of the decomposition, they do not uniquely specify the values of the different terms. Therefore, we investigate additional properties such as strong symmetry and left monotonicity. We find that strong symmetry is incompatible with the properties proposed by Williams and Beer. Although left monotonicity is a very natural property for an information measure it is not fulfilled by any of the proposed measures. We also study a geometric framework for information decompositions and ask whether it is possible to represent shared information by a family of posterior distributions. Finally, we draw connections to the notions of shared knowledge and common knowledge in game theory. While many people believe that independent variables cannot share information, we show that in game theory independent agents can have shared knowledge, but not common knowledge. We conclude that intuition and heuristic arguments do not suffice when arguing about information.

20 pages

References in corpus (2)

• Nonnegative Decomposition of Multivariate Information
• A Bivariate Measure of Redundant Information

Cited by in corpus (34)

• JIDT: An information-theoretic toolkit for studying the dynamics of complex systems
• Quantifying unique information
• Exploration of synergistic and redundant information sharing in static and dynamical Gaussian systems
• Efficient transfer entropy analysis of non-stationary neural time series
• Bits from Biology for Computational Intelligence
• Measuring Integrated Information: Comparison of Candidate Measures in Theory and Simulation
• Control of criticality and computation in spiking neuromorphic networks with plasticity
• Intersection Information based on Common Randomness
• Bits and Pieces: Understanding Information Decomposition from Part-whole Relationships and Formal Logic
• Pointwise Partial Information Decomposition using the Specificity and Ambiguity Lattices
• Multivariate Dependence Beyond Shannon Information
• A Novel Approach to the Partial Information Decomposition
• Invariant components of synergy, redundancy, and unique information among three variables
• Redundancy and synergy in dual decompositions of mutual information gain and information loss
• Introducing a differentiable measure of pointwise shared information
• Understanding interdependency through complex information sharing
• Quantifying Redundant Information in Predicting a Target Random Variable
• Quantifying how much sensory information in a neural code is relevant for behavior
• Decomposing past and future: Integrated information decomposition based on shared probability mass exclusions
• Reconsidering unique information: Towards a multivariate information decomposition
• On extractable shared information
• Probability Mass Exclusions and the Directed Components of Pointwise Mutual Information
• A partial information decomposition for discrete and continuous variables
• The identity of information: how deterministic dependencies constrain information synergy and redundancy
• Orders between channels and implications for partial information decomposition
• MAXENT3D_PID: An Estimator for the Maximum-entropy Trivariate Partial Information Decomposition
• Information Decomposition on Structured Space
• Explicit Formula for Partial Information Decomposition
• Stripping syntax from complexity: An information-theoretical perspective on complex systems
• Quantifying multivariate redundancy with maximum entropy decompositions of mutual information
• On shared and multiple information
• Some new insights into information decomposition in complex systems based on common information
• Partial Information Decomposition of Boolean Functions: a Fourier Analysis perspective
• An Approximation Scheme for Multivariate Information based on Partial Information Decomposition