A Short Note on the Jensen-Shannon Divergence between Simple Mixture Distributions

Abstract

This short note presents results about the symmetric Jensen-Shannon divergence between two discrete mixture distributions $p_1$ and $p_2$. Specifically, for $i=1,2$, $p_i$ is the mixture of a common distribution $q$ and a distribution ${\tilde{p}}_i$ with mixture proportion ${\lambda }_i$. In general, ${\tilde{p}}_1\ne {\tilde{p}}_2$ and ${\lambda }_1\ne {\lambda }_2$. We provide experimental and theoretical insight to the behavior of the symmetric Jensen-Shannon divergence between $p_1$ and $p_2$ as the mixture proportions or the divergence between ${\tilde{p}}_1$ and ${\tilde{p}}_2$ change. We also provide insight into scenarios where the supports of the distributions ${\tilde{p}}_1$, ${\tilde{p}}_2$, and $q$ do not coincide.

four-page tech note