Partial Information Decomposition via Deficiency for Multivariate Gaussians

Abstract

Bivariate partial information decompositions (PIDs) characterize how the information in a "message" random variable is decomposed between two "constituent" random variables in terms of unique, redundant and synergistic information components. These components are a function of the joint distribution of the three variables, and are typically defined using an optimization over the space of all possible joint distributions. This makes it computationally challenging to compute PIDs in practice and restricts their use to low-dimensional random vectors. To ease this burden, we consider the case of jointly Gaussian random vectors in this paper. This case was previously examined by Barrett (2015), who showed that certain operationally well-motivated PIDs reduce to a closed form expression for scalar messages. Here, we show that Barrett's result does not extend to vector messages in general, and characterize the set of multivariate Gaussian distributions that reduce to closed-form. Then, for all other multivariate Gaussian distributions, we propose a convex optimization framework for approximately computing a specific PID definition based on the statistical concept of deficiency. Using simplifying assumptions specific to the Gaussian case, we provide an efficient algorithm to approximately compute the bivariate PID for multivariate Gaussian variables with tens or even hundreds of dimensions. We also theoretically and empirically justify the goodness of this approximation.

arXiv commentPresented at ISIT 2022. This version has been updated to reflect the final conference publication, including appendices. It also corrects technical errors in Remark 1 and Appendix C, adds a new experiment, and has a substantially improved presentation as well as additional detail in the appendix, compared to the previous arxiv version

References in corpus (14)

• Estimating Mutual Information
• CVXPY: A Python-Embedded Modeling Language for Convex Optimization
• Quantifying unique information
• Nonnegative Decomposition of Multivariate Information
• Exploration of synergistic and redundant information sharing in static and dynamical Gaussian systems
• A Bivariate Measure of Redundant Information
• Pointwise Partial Information Decomposition using the Specificity and Ambiguity Lattices
• Comparison of Channels: Criteria for Domination by a Symmetric Channel
• Quantifying how much sensory information in a neural code is relevant for behavior
• Noisy-Interference Sum-Rate Capacity for Vector Gaussian Interference Channels
• Information Flow in Computational Systems
• Computing the Unique Information
• Exact partial information decompositions for Gaussian systems based on dependency constraints
• Unique Informations and Deficiencies

Cited by in corpus (1)

• Partial information decomposition: redundancy as information bottleneck