Estimating Mixture Entropy with Pairwise Distances

Abstract

Mixture distributions arise in many parametric and non-parametric settings -- for example, in Gaussian mixture models and in non-parametric estimation. It is often necessary to compute the entropy of a mixture, but, in most cases, this quantity has no closed-form expression, making some form of approximation necessary. We propose a family of estimators based on a pairwise distance function between mixture components, and show that this estimator class has many attractive properties. For many distributions of interest, the proposed estimators are efficient to compute, differentiable in the mixture parameters, and become exact when the mixture components are clustered. We prove this family includes lower and upper bounds on the mixture entropy. The Chernoff $\alpha$-divergence gives a lower bound when chosen as the distance function, with the Bhattacharyya distance providing the tightest lower bound for components that are symmetric and members of a location family. The Kullback-Leibler divergence gives an upper bound when used as the distance function. We provide closed-form expressions of these bounds for mixtures of Gaussians, and discuss their applications to the estimation of mutual information. We then demonstrate that our bounds are significantly tighter than well-known existing bounds using numeric simulations. This estimator class is very useful in optimization problems involving maximization/minimization of entropy and mutual information, such as MaxEnt and rate distortion problems.

Corrects several errata in published version, in particular in Section V (bounds on mutual information)

Cited by in corpus (17)

• Nonlinear Information Bottleneck
• Entropy and mutual information in models of deep neural networks
• On Information Plane Analyses of Neural Network Classifiers -- A Review
• HRel: Filter Pruning based on High Relevance between Activation Maps and Class Labels
• Mutual information disentangles interactions from changing environments
• The Convex Information Bottleneck Lagrangian
• Estimating the Mutual Information between two Discrete, Asymmetric Variables with Limited Samples
• Information flows of diverse autoencoders
• On the Achievable Rate and Capacity for a Sample-based Practical Photon-counting Receiver
• Error Bounds on a Mixed Entropy Inequality
• Bounds on mutual information of mixture data for classification tasks
• Inferring a property of a large system from a small number of samples
• An Approximate Bayesian Approach to Optimal Input Signal Design for System Identification
• Bounds on the rates of statistical divergences and mutual information via stochastic thermodynamics
• Harnessing Light for Cold-Start Recommendations: Leveraging Epistemic Uncertainty to Enhance Performance in User-Item Interactions
• Asymptotic Error Rates for Point Process Classification
• Mutual information in changing environments: non-linear interactions, out-of-equilibrium systems, and continuously-varying diffusivities