Entropy regularization in probabilistic clustering

Abstract

Bayesian nonparametric mixture models are widely used to cluster observations. However, one major drawback of the approach is that the estimated partition often presents unbalanced clusters' frequencies with only a few dominating clusters and a large number of sparsely-populated ones. This feature translates into results that are often uninterpretable unless we accept to ignore a relevant number of observations and clusters. Interpreting the posterior distribution as penalized likelihood, we show how the unbalance can be explained as a direct consequence of the cost functions involved in estimating the partition. In light of our findings, we propose a novel Bayesian estimator of the clustering configuration. The proposed estimator is equivalent to a post-processing procedure that reduces the number of sparsely-populated clusters and enhances interpretability. The procedure takes the form of entropy-regularization of the Bayesian estimate. While being computationally convenient with respect to alternative strategies, it is also theoretically justified as a correction to the Bayesian loss function used for point estimation and, as such, can be applied to any posterior distribution of clusters, regardless of the specific model used.

References in corpus (14)

• Bayesian cluster analysis: Point estimation and credible balls
• Discussion on Bayesian Cluster Analysis: Point Estimation and Credible Balls by Sara Wade and Zoubin Ghahramani
• Are Gibbs-type priors the most natural generalization of the Dirichlet process?
• Characteristics of Spiral Arms in Late-type Galaxies
• Clustering consistency with Dirichlet process mixtures
• Bayesian Inference for Latent Biologic Structure with Determinantal Point Processes (DPP)
• Flexible clustering via hidden hierarchical Dirichlet priors
• Centered Partition Process: Informative Priors for Clustering
• Spying on the prior of the number of data clusters and the partition distribution in Bayesian cluster analysis
• Flexible Models for Microclustering with Application to Entity Resolution
• Bayesian clustering of multiple zero-inflated outcomes
• Model Selection for Maternal Hypertensive Disorders with Symmetric Hierarchical Dirichlet Processes
• Why the Rich Get Richer? On the Balancedness of Random Partition Models
• Graph-Aligned Random Partition Model (GARP)

Cited by in corpus (1)

• Graph-Aligned Random Partition Model (GARP)