MAD-Bayes: MAP-based Asymptotic Derivations from Bayes

Abstract

The classical mixture of Gaussians model is related to K-means via small-variance asymptotics: as the covariances of the Gaussians tend to zero, the negative log-likelihood of the mixture of Gaussians model approaches the K-means objective, and the EM algorithm approaches the K-means algorithm. Kulis & Jordan (2012) used this observation to obtain a novel K-means-like algorithm from a Gibbs sampler for the Dirichlet process (DP) mixture. We instead consider applying small-variance asymptotics directly to the posterior in Bayesian nonparametric models. This framework is independent of any specific Bayesian inference algorithm, and it has the major advantage that it generalizes immediately to a range of models beyond the DP mixture. To illustrate, we apply our framework to the feature learning setting, where the beta process and Indian buffet process provide an appropriate Bayesian nonparametric prior. We obtain a novel objective function that goes beyond clustering to learn (and penalize new) groupings for which we relax the mutual exclusivity and exhaustivity assumptions of clustering. We demonstrate several other algorithms, all of which are scalable and simple to implement. Empirical results demonstrate the benefits of the new framework.

13 pages, 3 figures

References in corpus (1)

• Revisiting k-means: New Algorithms via Bayesian Nonparametrics

Cited by in corpus (19)

• Discussion on Bayesian Cluster Analysis: Point Estimation and Credible Balls by Sara Wade and Zoubin Ghahramani
• Bayesian cluster analysis: Point estimation and credible balls
• $k$-means as a variational EM approximation of Gaussian mixture models
• MAD Bayes for Tumor Heterogeneity Feature Allocation with Non-Normal Sampling
• Optimistic Concurrency Control for Distributed Unsupervised Learning
• Anchor & Transform: Learning Sparse Embeddings for Large Vocabularies
• Simple approximate MAP Inference for Dirichlet processes
• Generalized Dirichlet-process-means for $f$-separable distortion measures
• Combinatorial Topic Models using Small-Variance Asymptotics
• Consensus Monte Carlo for Random Subsets using Shared Anchors
• Accelerated Parallel Non-conjugate Sampling for Bayesian Non-parametric Models
• CRAFT: ClusteR-specific Assorted Feature selecTion
• TreeClone: Reconstruction of Tumor Subclone Phylogeny Based on Mutation Pairs using Next Generation Sequencing Data
• Efficient Tensor Decomposition with Boolean Factors
• Covariance Matrices for Mean Field Variational Bayes
• Parallel Markov Chain Monte Carlo for the Indian Buffet Process
• Controlling for sparsity in sparse factor analysis models: adaptive latent feature sharing for piecewise linear dimensionality reduction
• Detailed Derivations of Small-Variance Asymptotics for some Hierarchical Bayesian Nonparametric Models
• Small-Variance Asymptotics for Nonparametric Bayesian Overlapping Stochastic Blockmodels