Trimming the Independent Fat: Sufficient Statistics, Mutual Information, and Predictability from Effective Channel States
arXiv:1702.01831 · doi:10.1103/PhysRevE.95.060102
Abstract
One of the most fundamental questions one can ask about a pair of random variables X and Y is the value of their mutual information. Unfortunately, this task is often stymied by the extremely large dimension of the variables. We might hope to replace each variable by a lower-dimensional representation that preserves the relationship with the other variable. The theoretically ideal implementation is the use of minimal sufficient statistics, where it is well-known that either X or Y can be replaced by their minimal sufficient statistic about the other while preserving the mutual information. While intuitively reasonable, it is not obvious or straightforward that both variables can be replaced simultaneously. We demonstrate that this is in fact possible: the information X's minimal sufficient statistic preserves about Y is exactly the information that Y's minimal sufficient statistic preserves about X. As an important corollary, we consider the case where one variable is a stochastic process' past and the other its future and the present is viewed as a memoryful channel. In this case, the mutual information is the channel transmission rate between the channel's effective states. That is, the past-future mutual information (the excess entropy) is the amount of information about the future that can be predicted using the past. Translating our result about minimal sufficient statistics, this is equivalent to the mutual information between the forward- and reverse-time causal states of computational mechanics. We close by discussing multivariate extensions to this use of minimal sufficient statistics.
arXiv comment6 pages, 4 figures; http://csc.ucdavis.edu/~cmg/compmech/pubs/mcs.htm
References in corpus (10)
- Computational Mechanics: Pattern and Prediction, Structure and Simplicity
- Entropy and Long range correlations in literary English
- Entropy and Long range correlations in literary English
- Time's Barbed Arrow: Irreversibility, Crypticity, and Stored Information
- Statistical Complexity of Simple 1D Spin Systems
- Anatomy of a Bit: Information in a Time Series Observation
- Prediction, Retrodiction, and The Amount of Information Stored in the Present
- On the Vocabulary of Grammar-Based Codes and the Logical Consistency of Texts
- Information Symmetries in Irreversible Processes
- Statistical Signatures of Structural Organization: The case of long memory in renewal processes