Pooling Probability Distributions and the Partial Information Decomposition

Abstract

Notwithstanding various attempts to construct a Partial Information Decomposition (PID) for multiple variables by defining synergistic, redundant, and unique information, there is no consensus on how one ought to precisely define either of these quantities. One aim here is to illustrate how that ambiguity -- or, more positively, freedom of choice -- may arise. Using the basic idea that information equals the average reduction in uncertainty when going from an initial to a final probability distribution, synergistic information will likewise be defined as a difference between two entropies. One term is uncontroversial and characterizes ``the whole'' information that source variables carry jointly about a target variable $\varT$. The other term then is meant to characterize the information carried by the ``sum of its parts.'' Here we interpret that concept as needing a suitable probability distribution aggregated (``pooled'') from multiple marginal distributions (the parts). Ambiguity arises in the definition of the optimum way to pool two (or more) probability distributions. Independent of the exact definition of optimum pooling, the concept of pooling leads to a lattice that differs from the often-used redundancy-based lattice. One can associate not just a number (an average entropy) with each node of the lattice, but (pooled) probability distributions. As an example,one simple and reasonable approach to pooling is presented, which naturally gives rise to the overlap between different probability distributions as being a crucial quantity that characterizes both synergistic and unique information.

arXiv comment10 pages, submitted to Phys. Rev. E

References in corpus (14)

• The physics of higher-order interactions in complex systems
• Quantifying unique information
• Quantifying High-order Interdependencies via Multivariate Extensions of the Mutual Information
• A Bivariate Measure of Redundant Information
• Partial Information Decomposition as a Unified Approach to the Specification of Neural Goal Functions
• Measuring multivariate redundant information with pointwise common change in surprisal
• Bits and Pieces: Understanding Information Decomposition from Part-whole Relationships and Formal Logic
• Pointwise Partial Information Decomposition using the Specificity and Ambiguity Lattices
• Multivariate Dependence Beyond Shannon Information
• A Novel Approach to the Partial Information Decomposition
• An operational information decomposition via synergistic disclosure
• Secret Sharing and Shared Information
• Unique Information and Secret Key Agreement
• Combining probability distributions: Extending the logarithmic pooling approach