Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders

Abstract

We study a variant of the variational autoencoder model (VAE) with a Gaussian mixture as a prior distribution, with the goal of performing unsupervised clustering through deep generative models. We observe that the known problem of over-regularisation that has been shown to arise in regular VAEs also manifests itself in our model and leads to cluster degeneracy. We show that a heuristic called minimum information constraint that has been shown to mitigate this effect in VAEs can also be applied to improve unsupervised clustering performance with our model. Furthermore we analyse the effect of this heuristic and provide an intuition of the various processes with the help of visualizations. Finally, we demonstrate the performance of our model on synthetic data, MNIST and SVHN, showing that the obtained clusters are distinct, interpretable and result in achieving competitive performance on unsupervised clustering to the state-of-the-art results.

12 pages, 6 figures, Under review as a conference paper at ICLR 2017

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
• Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks
• Ladder Variational Autoencoders
• Attend, Infer, Repeat: Fast Scene Understanding with Generative Models
• Composing graphical models with neural networks for structured representations and fast inference
• Improving Variational Inference with Inverse Autoregressive Flow
• Unsupervised Deep Embedding for Clustering Analysis
• Stochastic Backpropagation through Mixture Density Distributions
• Tagger: Deep Unsupervised Perceptual Grouping

Cited by in corpus (142)

• Recent advances and clinical applications of deep learning in medical image analysis
• Learning Discrete Representations via Information Maximizing Self-Augmented Training
• SPICE: Semantic Pseudo-labeling for Image Clustering
• MR image reconstruction using deep density priors
• Unsupervised Detection of Lesions in Brain MRI using constrained adversarial auto-encoders
• SpectralNet: Spectral Clustering using Deep Neural Networks
• Unsupervised Learning for Cell-level Visual Representation in Histopathology Images with Generative Adversarial Networks
• Data Augmentation in High Dimensional Low Sample Size Setting Using a Geometry-Based Variational Autoencoder
• Hyperspherical Variational Auto-Encoders
• Graph Information Bottleneck
• Dual Adversarial Auto-Encoders for Clustering
• VAE with a VampPrior
• Topic-Guided Variational Autoencoders for Text Generation
• Disentanglement by Nonlinear ICA with General Incompressible-flow Networks (GIN)
• AutoEmbedder: A semi-supervised DNN embedding system for clustering
• The continuous Bernoulli: fixing a pervasive error in variational autoencoders
• Deblending galaxies with Variational Autoencoders: a joint multi-band, multi-instrument approach
• AstroVaDEr: Astronomical Variational Deep Embedder for Unsupervised Morphological Classification of Galaxies and Synthetic Image Generation
• Hierarchical Generative Modeling for Controllable Speech Synthesis
• Deep Discriminative Clustering Analysis
• Diverse Human Motion Prediction Guided by Multi-Level Spatial-Temporal Anchors
• Video Anomaly Detection and Localization via Gaussian Mixture Fully Convolutional Variational Autoencoder
• Deep Unsupervised Clustering Using Mixture of Autoencoders
• Clustering and Unsupervised Anomaly Detection with L2 Normalized Deep Auto-Encoder Representations
• Clustering-friendly Representation Learning via Instance Discrimination and Feature Decorrelation
• GAP: Generalizable Approximate Graph Partitioning Framework
• A Tutorial on Deep Latent Variable Models of Natural Language
• Stabilizing Invertible Neural Networks Using Mixture Models
• Auto-Encoding Total Correlation Explanation
• Diffusion Variational Autoencoders
• Hierarchical Quantized Autoencoders
• Learning Latent Representations in Neural Networks for Clustering through Pseudo Supervision and Graph-based Activity Regularization
• Poincaré Wasserstein Autoencoder
• Unsupervised Semantic Aggregation and Deformable Template Matching for Semi-Supervised Learning
• Variational Memory Addressing in Generative Models
• Coupled VAE: Improved Accuracy and Robustness of a Variational Autoencoder
• Learning Autoencoders with Relational Regularization
• Every Smile is Unique: Landmark-Guided Diverse Smile Generation
• Scalable Gaussian Process Variational Autoencoders
• RDEC: Integrating Regularization into Deep Embedded Clustering for Imbalanced Datasets
• An Explicit Local and Global Representation Disentanglement Framework with Applications in Deep Clustering and Unsupervised Object Detection
• Gaussian Mixture Variational Autoencoder with Contrastive Learning for Multi-Label Classification
• Degeneration in VAE: in the Light of Fisher Information Loss
• Latent space conditioning for improved classification and anomaly detection
• DLow: Diversifying Latent Flows for Diverse Human Motion Prediction
• Learning-Aided Physical Layer Attacks Against Multicarrier Communications in IoT
• Competitive Training of Mixtures of Independent Deep Generative Models
• X-ray Study of Spatial Structures in Tycho's Supernova Remnant Using Unsupervised Deep Learning
• GATCluster: Self-Supervised Gaussian-Attention Network for Image Clustering
• MiCE: Mixture of Contrastive Experts for Unsupervised Image Clustering
• Balancing Reconstruction Quality and Regularisation in ELBO for VAEs
• InfoCatVAE: Representation Learning with Categorical Variational Autoencoders
• Discriminative Hamiltonian Variational Autoencoder for Accurate Tumor Segmentation in Data-Scarce Regimes
• On the Necessity and Effectiveness of Learning the Prior of Variational Auto-Encoder
• Latent Simplex Position Model: High Dimensional Multi-view Clustering with Uncertainty Quantification
• BasisVAE: Translation-invariant feature-level clustering with Variational Autoencoders
• Identifying Mislabeled Images in Supervised Learning Utilizing Autoencoder
• Weakly Supervised Clustering by Exploiting Unique Class Count
• MMGAN: Generative Adversarial Networks for Multi-Modal Distributions
• Novelty Detection via Non-Adversarial Generative Network
• We are More than Our Joints: Predicting how 3D Bodies Move
• Gaussian mixture models with Wasserstein distance
• Pythae: Unifying Generative Autoencoders in Python -- A Benchmarking Use Case
• Geometry-Aware Hamiltonian Variational Auto-Encoder
• Generating Smooth Pose Sequences for Diverse Human Motion Prediction
• Learning Hierarchical Priors in VAEs
• Multi-objects Generation with Amortized Structural Regularization
• Variational Autoencoder with Implicit Optimal Priors
• Scalable Deep Unsupervised Clustering with Concrete GMVAEs
• Learning Disentangled Representations of Timbre and Pitch for Musical Instrument Sounds Using Gaussian Mixture Variational Autoencoders
• Congestion-aware Multi-agent Trajectory Prediction for Collision Avoidance
• A survey on Variational Autoencoders from a GreenAI perspective
• Meta-Reinforcement Learning in Broad and Non-Parametric Environments
• Improving Disentangled Representation Learning with the Beta Bernoulli Process
• Cross-Domain Medical Image Translation by Shared Latent Gaussian Mixture Model
• Lifelong Learning using Eigentasks: Task Separation, Skill Acquisition, and Selective Transfer
• Variational Feature Disentangling for Fine-Grained Few-Shot Classification
• Streaming Adaptive Nonparametric Variational Autoencoder
• Triggering Dark Showers with Conditional Dual Auto-Encoders
• Increasing Expressivity of a Hyperspherical VAE
• Unsupervised multi-modal Styled Content Generation
• Discond-VAE: Disentangling Continuous Factors from the Discrete
• Conditioning Latent-Space Clusters for Real-World Anomaly Classification
• Unsupervised Embedding of Hierarchical Structure in Euclidean Space
• Sum-Product-Transform Networks: Exploiting Symmetries using Invertible Transformations
• Continual Learning of New Sound Classes using Generative Replay
• Bigeminal Priors Variational auto-encoder
• Dissimilarity Mixture Autoencoder for Deep Clustering
• Singing Voice Conversion with Disentangled Representations of Singer and Vocal Technique Using Variational Autoencoders
• Truncated Gaussian-Mixture Variational AutoEncoder
• Deep Clustering With Intra-class Distance Constraint for Hyperspectral Images
• Inference and De-Noising of Non-Gaussian Particle Distribution Functions: A Generative Modeling Approach
• Unsupervised Star Galaxy Classification with Cascade Variational Auto-Encoder
• Likelihood Assignment for Out-of-Distribution Inputs in Deep Generative Models is Sensitive to Prior Distribution Choice
• Double cycle-consistent generative adversarial network for unsupervised conditional generation
• Deep Learning Based Unsupervised and Semi-supervised Classification for Keratoconus
• Automatic Differentiation Variational Inference with Mixtures
• Adversarially Learned Mixture Model
• A Framework for Joint Unsupervised Learning of Cluster-Aware Embedding for Heterogeneous Networks
• Dispersed Exponential Family Mixture VAEs for Interpretable Text Generation
• Multiview Variational Deep Learning with Application to Scalable Indoor Localization
• Variational Mixture of Normalizing Flows
• Simple, Scalable, and Stable Variational Deep Clustering
• InfoVAEGAN : learning joint interpretable representations by information maximization and maximum likelihood
• You Never Cluster Alone
• Automatic dimensionality selection for principal component analysis models with the ignorance score
• A Factorial Mixture Prior for Compositional Deep Generative Models
• Deep Unsupervised Clustering with Clustered Generator Model
• Multi-level Feature Learning on Embedding Layer of Convolutional Autoencoders and Deep Inverse Feature Learning for Image Clustering
• Bayesian neural networks and dimensionality reduction
• Density Deconvolution with Normalizing Flows
• Robust Self-Supervised Convolutional Neural Network for Subspace Clustering and Classification
• Deep Generative Pattern-Set Mixture Models for Nonignorable Missingness
• Cauchy-Schwarz Regularized Autoencoder
• Invariance-based Multi-Clustering of Latent Space Embeddings for Equivariant Learning
• Memory and attention in deep learning
• GMAIR: Unsupervised Object Detection Based on Spatial Attention and Gaussian Mixture
• Estimating the Robustness of Classification Models by the Structure of the Learned Feature-Space
• Self-Enhancing Multi-filter Sequence-to-Sequence Model
• DPR-CAE: Capsule Autoencoder with Dynamic Part Representation for Image Parsing
• Mixture Representation Learning with Coupled Autoencoders
• Defending Adversarial Attacks via Semantic Feature Manipulation
• Deep Generative Model with Beta Bernoulli Process for Modeling and Learning Confounding Factors
• Deep clustering of longitudinal data
• Supervised Encoding for Discrete Representation Learning
• MixPoet: Diverse Poetry Generation via Learning Controllable Mixed Latent Space
• Encoded Prior Sliced Wasserstein AutoEncoder for learning latent manifold representations
• Medical data wrangling with sequential variational autoencoders
• Deep Contrastive Patch-Based Subspace Learning for Camera Image Signal Processing
• EXoN: EXplainable encoder Network
• Coarse Grained Exponential Variational Autoencoders
• Variational embedding of protein folding simulations using gaussian mixture variational autoencoders
• Lifelong Mixture of Variational Autoencoders
• WiSE-ALE: Wide Sample Estimator for Approximate Latent Embedding
• A Conditional Generative Matching Model for Multi-lingual Reply Suggestion
• Multiclass non-Adversarial Image Synthesis, with Application to Classification from Very Small Sample
• Towards Stable Imbalanced Data Classification via Virtual Big Data Projection
• Improving Relational Regularized Autoencoders with Spherical Sliced Fused Gromov Wasserstein
• Generalized Clustering by Learning to Optimize Expected Normalized Cuts
• A Robust Speaker Clustering Method Based on Discrete Tied Variational Autoencoder
• Generating the support with extreme value losses
• Learning Priors for Adversarial Autoencoders