Exploration of synergistic and redundant information sharing in static and dynamical Gaussian systems

Abstract

To fully characterize the information that two `source' variables carry about a third `target' variable, one must decompose the total information into redundant, unique and synergistic components, i.e. obtain a partial information decomposition (PID). However Shannon's theory of information does not provide formulae to fully determine these quantities. Several recent studies have begun addressing this. Some possible definitions for PID quantities have been proposed, and some analyses have been carried out on systems composed of discrete variables. Here we present the first in-depth analysis of PIDs on Gaussian systems, both static and dynamical. We show that, for a broad class of Gaussian systems, previously proposed PID formulae imply that: (i) redundancy reduces to the minimum information provided by either source variable, and hence is independent of correlation between sources; (ii) synergy is the extra information contributed by the weaker source when the stronger source is known, and can either increase or decrease with correlation between sources. We find that Gaussian systems frequently exhibit net synergy, i.e. the information carried jointly by both sources is greater than the sum of informations carried by each source individually. Drawing from several explicit examples, we discuss the implications of these findings for measures of information transfer and information-based measures of complexity, both generally and within a neuroscience setting. Importantly, by providing independent formulae for synergy and redundancy applicable to continuous time-series data, we open up a new approach to characterizing and quantifying information sharing amongst complex system variables.

arXiv comment29 pages, 4 figures, in press Physical Review E, minor revisions to original version in response to peer review

References in corpus (15)

• Granger causality and transfer entropy are equivalent for Gaussian variables
• Quantifying unique information
• Nonnegative Decomposition of Multivariate Information
• Multivariate Granger Causality and Generalized Variance
• A Bivariate Measure of Redundant Information
• Differentiating information transfer and causal effect
• The functional structure of cortical neuronal networks grown in vitro
• Bits from Biology for Computational Intelligence
• Synergy and redundancy in the Granger causal analysis of dynamical networks
• Intersection Information based on Common Randomness
• Shared Information -- New Insights and Problems in Decomposing Information in Complex Systems
• Expanding the Transfer Entropy to Identify Information Subgraphs in Complex Systems
• On the Dynamical Complexity of Small-World Networks of Spiking Neurons
• Identification of functional information subgraphs in complex networks
• Redundant variables and Granger causality

Cited by in corpus (54)

• Quantifying High-order Interdependencies via Multivariate Extensions of the Mutual Information
• Partial Information Decomposition as a Unified Approach to the Specification of Neural Goal Functions
• Information Flows? A Critique of Transfer Entropies
• Reconciling emergences: An information-theoretic approach to identify causal emergence in multivariate data
• Bits from Biology for Computational Intelligence
• Measuring Integrated Information: Comparison of Candidate Measures in Theory and Simulation
• Multiscale Information Decomposition: Exact Computation for Multivariate Gaussian Processes
• Pointwise Partial Information Decomposition using the Specificity and Ambiguity Lattices
• Integrated information as a common signature of dynamical and information-processing complexity
• Quantifying information transfer and mediation along causal pathways in complex systems
• A Novel Approach to the Partial Information Decomposition
• Invariant components of synergy, redundancy, and unique information among three variables
• Redundancy and synergy in dual decompositions of mutual information gain and information loss
• An information-theoretic approach to self-organisation: Emergence of complex interdependencies in coupled dynamical systems
• An operational information decomposition via synergistic disclosure
• Understanding interdependency through complex information sharing
• Synergetic and redundant information flow detected by unnormalized Granger causality: application to resting state fMRI
• Dynamical independence: discovering emergent macroscopic processes in complex dynamical systems
• Towards an extended taxonomy of information dynamics via Integrated Information Decomposition
• Secret Sharing and Shared Information
• Quantifying how much sensory information in a neural code is relevant for behavior
• Decomposing past and future: Integrated information decomposition based on shared probability mass exclusions
• Generalised Measures of Multivariate Information Content
• Quantifying high-order interdependencies on individual patterns via the local O-information: theory and applications to music analysis
• Synergy as a warning sign of transitions: the case of the two-dimensional Ising model
• Redundancy in the information transmission in a two-step cascade
• Exact partial information decompositions for Gaussian systems based on dependency constraints
• The Partial Entropy Decomposition: Decomposing multivariate entropy and mutual information via pointwise common surprisal
• Integrated Information in the Spiking-Bursting Stochastic Model
• An information-theoretic approach to hypergraph psychometrics
• Synergistic information in a dynamical model implemented on the human structural connectome reveals spatially distinct associations with age
• A Framework for Causal Discovery in non-intervenable systems
• Assessing high-order effects in feature importance via predictability decomposition
• A partial information decomposition for discrete and continuous variables
• Partial Information Decomposition via Deficiency for Multivariate Gaussians
• The identity of information: how deterministic dependencies constrain information synergy and redundancy
• The coherent feed-forward loop acts as an efficient information transmitting motif
• Synergy, Redundancy and Common Information
• Interplay of synergy and redundancy in diamond motif
• Information processing in a simple one-step cascade
• Orders between channels and implications for partial information decomposition
• Integrated Information Decomposition Unveils Major Structural Traits of $In$ $Silico$ and $In$ $Vitro$ Neuronal Networks
• MAXENT3D_PID: An Estimator for the Maximum-entropy Trivariate Partial Information Decomposition
• A Measure of Synergy based on Union Information
• Causal influence in linear response models
• Quantifying multivariate redundancy with maximum entropy decompositions of mutual information
• Disentanglement Analysis with Partial Information Decomposition
• Multivariate Partial Information Decomposition: Constructions, Inconsistencies, and Alternative Measures
• Information Synergy in the Anticipatory Dynamics of a Retina
• Multiscale Partial Information Decomposition of Dynamic Processes with Short and Long-range correlations: Theory and Application to Cardiovascular Control
• Partial Information Decomposition of Boolean Functions: a Fourier Analysis perspective
• Information transmission in a two-step cascade: Interplay of activation and repression
• An Approximation Scheme for Multivariate Information based on Partial Information Decomposition
• Composite Gaussian Processes: Scalable Computation and Performance Analysis