Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models

Abstract

Learning generative models that span multiple data modalities, such as vision and language, is often motivated by the desire to learn more useful, generalisable representations that faithfully capture common underlying factors between the modalities. In this work, we characterise successful learning of such models as the fulfillment of four criteria: i) implicit latent decomposition into shared and private subspaces, ii) coherent joint generation over all modalities, iii) coherent cross-generation across individual modalities, and iv) improved model learning for individual modalities through multi-modal integration. Here, we propose a mixture-of-experts multimodal variational autoencoder (MMVAE) to learn generative models on different sets of modalities, including a challenging image-language dataset, and demonstrate its ability to satisfy all four criteria, both qualitatively and quantitatively.

Cited by in corpus (18)

• A survey of multimodal deep generative models
• Autoencoding with a Classifier System
• In-Bed Human Pose Estimation from Unseen and Privacy-Preserving Image Domains
• Unified Brain MR-Ultrasound Synthesis using Multi-Modal Hierarchical Representations
• Gaussian Mixture Variational Autoencoder with Contrastive Learning for Multi-Label Classification
• lamBERT: Language and Action Learning Using Multimodal BERT
• Fork or Fail: Cycle-Consistent Training with Many-to-One Mappings
• Cross-modal Variational Auto-encoder for Content-based Micro-video Background Music Recommendation
• Capturing Label Characteristics in VAEs
• Multimodal Variational Autoencoders for Semi-Supervised Learning: In Defense of Product-of-Experts
• MHVAE: a Human-Inspired Deep Hierarchical Generative Model for Multimodal Representation Learning
• Learning Multimodal VAEs through Mutual Supervision
• Learning View-Disentangled Human Pose Representation by Contrastive Cross-View Mutual Information Maximization
• On the Limitations of Multimodal VAEs
• Unsupervised Domain Adaptation with Variational Approximation for Cardiac Segmentation
• A Conditional Generative Matching Model for Multi-lingual Reply Suggestion
• Lifelong Mixture of Variational Autoencoders
• Controlling the Interaction Between Generation and Inference in Semi-Supervised Variational Autoencoders Using Importance Weighting