Synergy, Redundancy and Common Information
arXiv:1509.03706
Abstract
We consider the problem of decomposing the total mutual information conveyed by a pair of predictor random variables about a target random variable into redundant, unique and synergistic contributions. We focus on the relationship between "redundant information" and the more familiar information-theoretic notions of "common information". Our main contribution is an impossibility result. We show that for independent predictor random variables, any common information based measure of redundancy cannot induce a nonnegative decomposition of the total mutual information. Interestingly, this entails that any reasonable measure of redundant information cannot be derived by optimization over a single random variable.
arXiv comment16 pages, 3 figures
References in corpus (10)
- The information bottleneck method
- Network information and connected correlations
- Quantifying unique information
- Nonnegative Decomposition of Multivariate Information
- Exploration of synergistic and redundant information sharing in static and dynamical Gaussian systems
- A Bivariate Measure of Redundant Information
- A Nonstochastic Information Theory for Communication and State Estimation
- Intersection Information based on Common Randomness
- Complexity Measures from Interaction Structures
- Quantifying Redundant Information in Predicting a Target Random Variable
Cited by in corpus (4)
- Redundancy and synergy in dual decompositions of mutual information gain and information loss
- The Partial Entropy Decomposition: Decomposing multivariate entropy and mutual information via pointwise common surprisal
- Quantifying multivariate redundancy with maximum entropy decompositions of mutual information
- Extended Gray-Wyner System with Complementary Causal Side Information