A uniqueness theorem for the variational free energy decomposition

Abstract

For a finite system with reference measure $p$ and positive weight $\ell$, the variational free energy $F(Q;p,\ell )=D(Q\Vert p)-E_Q[\log \ell ]$ satisfies the exact identity $\log Z(p,\ell )=-F(Q;p,\ell )+D(Q\Vert \ddot{I}€)$, where $Z$ is the partition function and $\ddot{I}€$ the associated Gibbs measure. For a uniform reference measure this is the Gibbs-Bogoliubov inequality of mean-field theory; for a Bayesian model it is the evidence decomposition of variational inference. Variational objectives built from $\hat{I}\pm$- or Rényi divergences retain useful bounds on $\log Z$ but not identities of this form, which raises the question of which functionals admit an exact decomposition. We prove the following characterization. Suppose a pair $(G,\hat{I}”)$ satisfies $\log Z=-G+\hat{I}”$ with $\hat{I}”$ a function of $Q$ and $\ddot{I}€$ alone; suppose $G$ is additively separable into a term depending on the reference measure and a term depending on the weight, with mild regularity; and suppose $\hat{I}”$ is nonnegative and vanishes precisely at $Q=\ddot{I}€$. Then $G=F$ and $\hat{I}”=D(\cdot \Vert \ddot{I}€)$. The proof reduces the hypotheses to a homomorphism from the multiplicative group of positive functions into $(\mathbb{R},+)$ and removes it using the second-order vanishing of relative entropy at its minimum. Counterexamples show that each hypothesis is needed. The result characterizes the decomposition rather than the divergence, and is therefore complementary to the axiomatic characterizations of relative entropy due to Shore-Johnson, Csiszár and Amari, which it neither uses nor extends.

11 pages