Reconsidering unique information: Towards a multivariate information decomposition

Abstract

The information that two random variables $Y$, $Z$ contain about a third random variable $X$ can have aspects of shared information (contained in both $Y$ and $Z$), of complementary information (only available from $(Y,Z)$ together) and of unique information (contained exclusively in either $Y$ or $Z$). Here, we study measures $\widetilde{SI}$ of shared, $\widetilde{UI}$ unique and $\widetilde{CI}$ complementary information introduced by Bertschinger et al., which are motivated from a decision theoretic perspective. We find that in most cases the intuitive rule that more variables contain more information applies, with the exception that $\widetilde{SI}$ and $\widetilde{CI}$ information are not monotone in the target variable $X$. Additionally, we show that it is not possible to extend the bivariate information decomposition into $\widetilde{SI}$, $\widetilde{UI}$ and $\widetilde{CI}$ to a non-negative decomposition on the partial information lattice of Williams and Beer. Nevertheless, the quantities $\widetilde{UI}$, $\widetilde{SI}$ and $\widetilde{CI}$ have a well-defined interpretation, even in the multivariate setting.

5 pages, 1 figure, submitted to ISIT 2014