-MLE: A fast algorithm for learning statistical mixture models
arXiv:1203.5181 · doi:10.1109/ICASSP.2012.6288022
Abstract
We describe -MLE, a fast and efficient local search algorithm for learning finite statistical mixtures of exponential families such as Gaussian mixture models. Mixture models are traditionally learned using the expectation-maximization (EM) soft clustering technique that monotonically increases the incomplete (expected complete) likelihood. Given prescribed mixture weights, the hard clustering -MLE algorithm iteratively assigns data to the most likely weighted component and update the component models using Maximum Likelihood Estimators (MLEs). Using the duality between exponential families and Bregman divergences, we prove that the local convergence of the complete likelihood of -MLE follows directly from the convergence of a dual additively weighted Bregman hard clustering. The inner loop of -MLE can be implemented using any -means heuristic like the celebrated Lloyd's batched or Hartigan's greedy swap updates. We then show how to update the mixture weights by minimizing a cross-entropy criterion that implies to update weights by taking the relative proportion of cluster points, and reiterate the mixture parameter update and mixture weight update processes until convergence. Hard EM is interpreted as a special case of -MLE when both the component update and the weight update are performed successively in the inner loop. To initialize -MLE, we propose -MLE++, a careful initialization of -MLE guaranteeing probabilistically a global bound on the best possible complete likelihood.
arXiv comment31 pages, Extend preliminary paper presented at IEEE ICASSP 2012
References in corpus (6)
- Polynomial Learning of Distribution Families
- Statistical exponential families: A digest with flash cards
- Bregman Voronoi Diagrams: Properties, Algorithms and Applications
- Inference and Evaluation of the Multinomial Mixture Model for Text Clustering
- Penalized maximum likelihood for multivariate Gaussian mixture
- Approximate Bregman near neighbors in sublinear time: Beyond the triangle inequality
Cited by in corpus (4)
- Fast approximations of the Jeffreys divergence between univariate Gaussian mixture models via exponential polynomial densities
- On -mixtures: Finite convex combinations of prescribed component distributions
- Estimation of multivariate generalized gamma convolutions through Laguerre expansions
- Convex Relaxations of Bregman Divergence Clustering