The Boltzmann/Shannon entropy as a measure of correlation

Abstract

IIt is demonstrated that the entropy of statistical mechanics and of information theory, $S(\mathbf{p})=-\sum p_i\log p_i$ may be viewed as a measure of correlation. Given a probability distribution on two discrete variables, $p_{ij}$, we define the correlation-destroying transformation $C:p_{ij}\to {\pi }_{ij}$, which creates a new distribution on those same variables in which no correlation exists between the variables, i.e. ${\pi }_{ij}=P_i{Q}_j$. It is then shown that the entropy obeys the relation $S(\mathbf{p})\le S(\pi )=S(\mathbf{P})+S(\mathbf{Q})$, i.e. the entropy is non-decreasing under these correlation-destroying transformations.