Nonnegative Decomposition of Multivariate Information

Abstract

Of the various attempts to generalize information theory to multiple variables, the most widely utilized, interaction information, suffers from the problem that it is sometimes negative. Here we reconsider from first principles the general structure of the information that a set of sources provides about a given variable. We begin with a new definition of redundancy as the minimum information that any source provides about each possible outcome of the variable, averaged over all possible outcomes. We then show how this measure of redundancy induces a lattice over sets of sources that clarifies the general structure of multivariate information. Finally, we use this redundancy lattice to propose a definition of partial information atoms that exhaustively decompose the Shannon information in a multivariate system in terms of the redundancy between synergies of subsets of the sources. Unlike interaction information, the atoms of our partial information decomposition are never negative and always support a clear interpretation as informational quantities. Our analysis also demonstrates how the negativity of interaction information can be explained by its confounding of redundancy and synergy.

14 pages, 9 figures

Cited by in corpus (14)

• JIDT: An information-theoretic toolkit for studying the dynamics of complex systems
• Exploration of synergistic and redundant information sharing in static and dynamical Gaussian systems
• Discovering Structure in High-Dimensional Data Through Correlation Explanation
• Quantifying how much sensory information in a neural code is relevant for behavior
• Secret Sharing and Shared Information
• Maximally Informative Hierarchical Representations of High-Dimensional Data
• Estimating the Mutual Information between two Discrete, Asymmetric Variables with Limited Samples
• The Partial Entropy Decomposition: Decomposing multivariate entropy and mutual information via pointwise common surprisal
• On active information storage in input-driven systems
• Is Consciousness Computable? Quantifying Integrated Information Using Algorithmic Information Theory
• Multiscale Structure of More-than-Binary Variables
• Some new insights into information decomposition in complex systems based on common information
• An Approximation Scheme for Multivariate Information based on Partial Information Decomposition
• Correlated structural evolution within multiplex networks