Nonnegative Decomposition of Multivariate Information

Abstract

Of the various attempts to generalize information theory to multiple variables, the most widely utilized, interaction information, suffers from the problem that it is sometimes negative. Here we reconsider from first principles the general structure of the information that a set of sources provides about a given variable. We begin with a new definition of redundancy as the minimum information that any source provides about each possible outcome of the variable, averaged over all possible outcomes. We then show how this measure of redundancy induces a lattice over sets of sources that clarifies the general structure of multivariate information. Finally, we use this redundancy lattice to propose a definition of partial information atoms that exhaustively decompose the Shannon information in a multivariate system in terms of the redundancy between synergies of subsets of the sources. Unlike interaction information, the atoms of our partial information decomposition are never negative and always support a clear interpretation as informational quantities. Our analysis also demonstrates how the negativity of interaction information can be explained by its confounding of redundancy and synergy.

14 pages, 9 figures

Cited by in corpus (75)

• JIDT: An information-theoretic toolkit for studying the dynamics of complex systems
• Quantifying unique information
• Exploration of synergistic and redundant information sharing in static and dynamical Gaussian systems
• Quantifying High-order Interdependencies via Multivariate Extensions of the Mutual Information
• A Bivariate Measure of Redundant Information
• Partial Information Decomposition as a Unified Approach to the Specification of Neural Goal Functions
• Information Flows? A Critique of Transfer Entropies
• Reconciling emergences: An information-theoretic approach to identify causal emergence in multivariate data
• Efficient transfer entropy analysis of non-stationary neural time series
• Bits from Biology for Computational Intelligence
• Anatomy of a Bit: Information in a Time Series Observation
• Intersection Information based on Common Randomness
• Expanding the Transfer Entropy to Identify Information Subgraphs in Complex Systems
• Shared Information -- New Insights and Problems in Decomposing Information in Complex Systems
• Bits and Pieces: Understanding Information Decomposition from Part-whole Relationships and Formal Logic
• Discovering Structure in High-Dimensional Data Through Correlation Explanation
• A framework for the local information dynamics of distributed computation in complex systems
• Redundancy and synergy in dual decompositions of mutual information gain and information loss
• Introducing a differentiable measure of pointwise shared information
• Understanding interdependency through complex information sharing
• Dynamical independence: discovering emergent macroscopic processes in complex dynamical systems
• Synergetic and redundant information flow detected by unnormalized Granger causality: application to resting state fMRI
• Towards an extended taxonomy of information dynamics via Integrated Information Decomposition
• Quantifying Redundant Information in Predicting a Target Random Variable
• Information flow through a model of the C. elegans klinotaxis circuit
• Secret Sharing and Shared Information
• Beyond integrated information: A taxonomy of information dynamics phenomena
• Quantifying how much sensory information in a neural code is relevant for behavior
• Understanding Neural Networks and Individual Neuron Importance via Information-Ordered Cumulative Ablation
• Maximally Informative Hierarchical Representations of High-Dimensional Data
• Quantifying high-order interdependencies on individual patterns via the local O-information: theory and applications to music analysis
• Information Flow in Computational Systems
• Estimating the Mutual Information between two Discrete, Asymmetric Variables with Limited Samples
• A Graph Algorithmic Approach to Separate Direct from Indirect Neural Interactions
• Biological Information
• The Partial Entropy Decomposition: Decomposing multivariate entropy and mutual information via pointwise common surprisal
• Understanding Convolutional Neural Networks with Information Theory: An Initial Exploration
• The Elusive Present: Hidden Past and Future Dependency and Why We Build Models
• A Framework for Causal Discovery in non-intervenable systems
• Direct and Indirect Effects -- An Information Theoretic Perspective
• A partial information decomposition for discrete and continuous variables
• Hidden Structures of Information Transport Underlying Spiral Wave Dynamics
• Partial Information Decomposition via Deficiency for Multivariate Gaussians
• Towards a Framework for Observational Causality From Time Series: When Shannon Meets Turing
• Decomposing information into copying versus transformation
• Synergy, Redundancy and Common Information
• On active information storage in input-driven systems
• Interplay of synergy and redundancy in diamond motif
• A Rigorous Information-Theoretic Definition of Redundancy and Relevancy in Feature Selection Based on (Partial) Information Decomposition
• Information dynamics of price and liquidity around the 2017 Bitcoin markets crash
• Elements of Consciousness and Cognition. Biology, Mathematic, Physics and Panpsychism: an Information Topology Perspective
• Contrasting information theoretic decompositions of modulatory and arithmetic interactions in neural information processing systems
• Is Consciousness Computable? Quantifying Integrated Information Using Algorithmic Information Theory
• Properties of Unique Information
• Quantifying multivariate redundancy with maximum entropy decompositions of mutual information
• Information encryption in the expert management of strategic uncertainty
• Stripping syntax from complexity: An information-theoretical perspective on complex systems
• On shared and multiple information
• Information Theoretic Representation Distillation
• Estimating informativeness of samples with Smooth Unique Information
• On Information Links
• Intersectional synergies: untangling irreducible effects of intersecting identities via information decomposition
• Neural Dependency Coding inspired Multimodal Fusion
• Information theoretic analysis of computational models as a tool to understand the neural basis of behaviors
• Multiscale Structure of More-than-Binary Variables
• Disentanglement Analysis with Partial Information Decomposition
• Some new insights into information decomposition in complex systems based on common information
• An Approximation Scheme for Multivariate Information based on Partial Information Decomposition
• A Generative Word Embedding Model and its Low Rank Positive Semidefinite Solution
• Iterative Scaling Algorithm for Channels
• Information transmission in a two-step cascade: Interplay of activation and repression
• Irreducibility is Minimum Synergy Among Parts
• Examining the causal structures of deep neural networks using information theory
• Partial Information Decomposition of Boolean Functions: a Fourier Analysis perspective
• Correlated structural evolution within multiplex networks