Multivariate Dependence Beyond Shannon Information

Abstract

Accurately determining dependency structure is critical to discovering a system's causal organization. We recently showed that the transfer entropy fails in a key aspect of this---measuring information flow---due to its conflation of dyadic and polyadic relationships. We extend this observation to demonstrate that this is true of all such Shannon information measures when used to analyze multivariate dependencies. This has broad implications, particularly when employing information to express the organization and mechanisms embedded in complex systems, including the burgeoning efforts to combine complex network theory with information theory. Here, we do not suggest that any aspect of information theory is wrong. Rather, the vast majority of its informational measures are simply inadequate for determining the meaningful dependency structure within joint probability distributions. Therefore, such information measures are inadequate for discovering intrinsic causal relations. We close by demonstrating that such distributions exist across an arbitrary set of variables.

10 pages, 6 figures, 3 tables; http://csc.ucdavis.edu/~cmg/compmech/pubs/mdbsi.htm

References in corpus (15)

• Local information transfer as a spatiotemporal filter for complex systems
• Dynamic information routing in complex networks
• Causation Entropy Identifies Indirect Influences, Dominance of Neighbors and Anticipatory Couplings
• The functional structure of cortical neuronal networks grown in vitro
• Information Flows? A Critique of Transfer Entropies
• Measuring multivariate redundant information with pointwise common change in surprisal
• Model-free quantification of time-series predictability
• Invariant components of synergy, redundancy, and unique information among three variables
• Identification of functional information subgraphs in complex networks
• On the sufficiency of pairwise interactions in maximum entropy models of biological networks
• Understanding interdependency through complex information sharing
• What did Erwin Mean? The Physics of Information from the Materials Genomics of Aperiodic Crystals and Water to Molecular Information Catalysts and Life
• Information flow through a model of the C. elegans klinotaxis circuit
• An Information-Theoretic Formalism for Multiscale Structure in Complex Systems
• Information Theoretic Approach to Social Networks

Cited by in corpus (28)

• Quantifying High-order Interdependencies via Multivariate Extensions of the Mutual Information
• Measuring multivariate redundant information with pointwise common change in surprisal
• Pointwise Partial Information Decomposition using the Specificity and Ambiguity Lattices
• How causal analysis can reveal autonomy in models of biological systems
• Invariant components of synergy, redundancy, and unique information among three variables
• Redundancy and synergy in dual decompositions of mutual information gain and information loss
• Generalised Measures of Multivariate Information Content
• Local Causal States and Discrete Coherent Structures
• Fraudulent White Noise: Flat power spectra belie arbitrarily complex processes
• The Partial Entropy Decomposition: Decomposing multivariate entropy and mutual information via pointwise common surprisal
• An information-theoretic approach to hypergraph psychometrics
• Causal network discovery by iterative conditioning: comparison of algorithms
• A Framework for Causal Discovery in non-intervenable systems
• infotheory: A C++/Python package for multivariate information theoretic analysis
• Assessing Information Transmission in Data Transformations with the Channel Multivariate Entropy Triangle
• Towards a Framework for Observational Causality From Time Series: When Shannon Meets Turing
• The identity of information: how deterministic dependencies constrain information synergy and redundancy
• Pooling Probability Distributions and the Partial Information Decomposition
• Higher-Order Interactions and Their Duals Reveal Synergy and Logical Dependence beyond Shannon-Information
• MAXENT3D_PID: An Estimator for the Maximum-entropy Trivariate Partial Information Decomposition
• Quantifying multivariate redundancy with maximum entropy decompositions of mutual information
• On shared and multiple information
• The dynamic importance of nodes is poorly predicted by static network features
• Multiscale Structure of More-than-Binary Variables
• Neural Dependency Coding inspired Multimodal Fusion
• Correlated structural evolution within multiplex networks
• Improving Multimodal fusion via Mutual Dependency Maximisation
• The Poincaré-Boltzmann Machine: from Statistical Physics to Machine Learning and back