Higher-order Common Information

Abstract

Shannon's mutual information quantifies dependence between two random variables. We introduce \emph{higher-order common information} (HCI), a measure of statistical relevance across $n$ random variables. HCI is defined through a sequential information-bottleneck construction in which an auxiliary representation is generated locally from a single reference source and successively optimized for relevance to the remaining sources. The terminal representation is evaluated by its minimum mutual information with any individual source. Thus, an HCI value of $r$ guarantees a locally generated representation having at least $r$ bits of statistical information about every source. We derive closed-form expressions for jointly Gaussian sources and for a binary common-source model for arbitrary finite $n$. For finite-alphabet sources, we further show that the HCI is lower-bounded by the Gács--Körner common information, while it is upper-bounded by the smallest pairwise mutual information. Finally, we propose an HCI-inspired numerical approximation and illustrate its application to multivariate EEG data.

This is the final preprint version