Multimodal Generative Models for Compositional Representation Learning
arXiv:1912.05075
Abstract
As deep neural networks become more adept at traditional tasks, many of the most exciting new challenges concern multimodality---observations that combine diverse types, such as image and text. In this paper, we introduce a family of multimodal deep generative models derived from variational bounds on the evidence (data marginal likelihood). As part of our derivation we find that many previous multimodal variational autoencoders used objectives that do not correctly bound the joint marginal likelihood across modalities. We further generalize our objective to work with several types of deep generative model (VAE, GAN, and flow-based), and allow use of different model types for different modalities. We benchmark our models across many image, label, and text datasets, and find that our multimodal VAEs excel with and without weak supervision. Additional improvements come from use of GAN image models with VAE language models. Finally, we investigate the effect of language on learned image representations through a variety of downstream tasks, such as compositionally, bounding box prediction, and visual relation prediction. We find evidence that these image representations are more abstract and compositional than equivalent representations learned from only visual data.
24 pages content; 7 pages appendix
References in corpus (24)
- Adam: A Method for Stochastic Optimization
- Conditional Generative Adversarial Nets
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- YOLOv3: An Incremental Improvement
- Stochastic Backpropagation and Approximate Inference in Deep Generative Models
- Microsoft COCO Captions: Data Collection and Evaluation Server
- NICE: Non-linear Independent Components Estimation
- Density estimation using Real NVP
- Adversarially Learned Inference
- Autoencoding beyond pixels using a learned similarity metric
- Invertible Conditional GANs for image editing
- InfoVAE: Information Maximizing Variational Autoencoders
- Learn to Combine Modalities in Multimodal Deep Learning
- Towards Deeper Understanding of Variational Autoencoding Models
- FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models
- Joint Multimodal Learning with Deep Generative Models
- Deep Variational Canonical Correlation Analysis
- Improving Variational Auto-Encoders using Householder Flow
- Generative Models of Visually Grounded Imagination
- ShapeWorld - A new test methodology for multimodal language understanding
- Measuring Compositionality in Representation Learning
- Factorized Inference in Deep Markov Models for Incomplete Multimodal Time Series