On the Limitations of Multimodal VAEs

Abstract

Multimodal variational autoencoders (VAEs) have shown promise as efficient generative models for weakly-supervised data. Yet, despite their advantage of weak supervision, they exhibit a gap in generative quality compared to unimodal VAEs, which are completely unsupervised. In an attempt to explain this gap, we uncover a fundamental limitation that applies to a large family of mixture-based multimodal VAEs. We prove that the sub-sampling of modalities enforces an undesirable upper bound on the multimodal ELBO and thereby limits the generative quality of the respective models. Empirically, we showcase the generative quality gap on both synthetic and real data and present the tradeoffs between different variants of multimodal VAEs. We find that none of the existing approaches fulfills all desired criteria of an effective multimodal generative model when applied on more complex datasets than those used in previous benchmarks. In summary, we identify, formalize, and validate fundamental limitations of VAE-based approaches for modeling weakly-supervised data and discuss implications for real-world applications.

arXiv commentICLR 2022 camera-ready version

References in corpus (12)

• Auto-Encoding Variational Bayes
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Zero-Shot Text-to-Image Generation
• Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
• Learning Factorized Multimodal Representations
• On Variational Bounds of Mutual Information
• Hetero-Modal Variational Encoder-Decoder for Joint Modality Completion and Segmentation
• Joint Multimodal Learning with Deep Generative Models
• Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models
• DIVA: Domain Invariant Variational Autoencoders
• M6: A Chinese Multimodal Pretrainer
• A Variational Information Bottleneck Approach to Multi-Omics Data Integration