The many faces of multivariate information

Abstract

Extracting higher-order structures from multivariate data has become an area of intensive study in complex systems science, as these multipartite interactions can reveal insights into fundamental features of complex systems like emergent phenomena. Information theory provides a natural language for exploring these interactions, as it elegantly formalizes the problem of comparing "wholes" and "parts" using joint, conditional, and marginal entropies. A large number of distinct statistics have been developed over the years, all aiming to capture different aspects of "higher-order" information sharing. Here, we show that these functions are special cases of a more general function, $\hat{I}{”}^k$ which is parameterized by a free parameter $k$. Generally, the $\hat{I}{”}^k$ function is arranged into a hierarchy of increasingly high-order synergies; for a given value of $k$, if $\hat{I}{”}^k(\mathbf{X})>0$, then $\mathbf{X}$ is dominated by interactions with order greater than $k$, while if $\hat{I}{”}^k(\mathbf{X})<0$, then $\mathbf{X}$ is dominated by interactions with order lower than $k$. Using the entropic conjugation framework, we also find that the conjugate of $\hat{I}{”}^k$, which we term $\hat{I}{“}^k$ is arranged into a similar hierarchy of increasingly high-order redundancies. Finally, we show that the interpretation of $\hat{I}{”}^k$ as a measure of synergy is combinatorial, rather than specific to any particular information-theoretic measure, allowing us to generalize the whole framework and define measures of synergy on any set function that meets certain criteria. Using the graph cyclomatic number as a case study, we derive topological analogues of the dual total correlation, O-information, and S-information that describe the cyclic structure of simple graphs.