Multivariate Partial Information Decomposition: Constructions, Inconsistencies, and Alternative Measures

Abstract

While mutual information effectively quantifies dependence between two variables, it does not by itself reveal the complex, fine-grained interactions among variables, i.e., how multiple sources contribute redundantly, uniquely, or synergistically to a target in multivariate settings. The Partial Information Decomposition (PID) framework was introduced to address this by decomposing the mutual information between a set of source variables and a target variable into fine-grained information atoms such as redundant, unique, and synergistic components. In this work, we review the axiomatic system and desired properties of the PID framework and make three main contributions. First, we resolve the two-source PID case by providing explicit closed-form formulas for all information atoms that satisfy the full set of axioms and desirable properties. Second, we prove that for three or more sources, PID suffers from fundamental inconsistencies: we review the known three-variable counterexample where the sum of atoms exceeds the total information, and extend it to a comprehensive impossibility theorem showing that no lattice-based decomposition can be consistent for all subsets when the number of sources exceeds three. Finally, we deviate from the PID lattice approach to avoid its inconsistencies, and present explicit measures of multivariate unique and synergistic information. Our proposed measures, which rely on new systems of random variables that eliminate higher-order dependencies, satisfy key axioms such as additivity and continuity, provide a robust theoretical explanation of high-order relations, and show strong numerical performance in comprehensive experiments on the Ising model. Our findings highlight the need for a new framework for studying multivariate information decomposition.

References in corpus (13)

• Quantifying unique information
• Exploration of synergistic and redundant information sharing in static and dynamical Gaussian systems
• A Bivariate Measure of Redundant Information
• Reconciling emergences: An information-theoretic approach to identify causal emergence in multivariate data
• Measuring multivariate redundant information with pointwise common change in surprisal
• Intersection Information based on Common Randomness
• Bits and Pieces: Understanding Information Decomposition from Part-whole Relationships and Formal Logic
• Pointwise Partial Information Decomposition using the Specificity and Ambiguity Lattices
• A Novel Approach to the Partial Information Decomposition
• Redundancy and synergy in dual decompositions of mutual information gain and information loss
• An operational information decomposition via synergistic disclosure
• Fractal Structure of Spin Clusters and Domain Walls in 2D Ising Model
• Generalised Measures of Multivariate Information Content