Multimodal Generative Models for Scalable Weakly-Supervised Learning

Abstract

Multiple modalities often co-occur when describing natural phenomena. Learning a joint representation of these modalities should yield deeper and more useful representations. Previous generative approaches to multi-modal input either do not learn a joint distribution or require additional computation to handle missing data. Here, we introduce a multimodal variational autoencoder (MVAE) that uses a product-of-experts inference network and a sub-sampled training paradigm to solve the multi-modal inference problem. Notably, our model shares parameters to efficiently learn under any combination of missing modalities. We apply the MVAE on four datasets and match state-of-the-art performance using many fewer parameters. In addition, we show that the MVAE is directly applicable to weakly-supervised learning, and is robust to incomplete supervision. We then consider two case studies, one of learning image transformations---edge detection, colorization, segmentation---as a set of modalities, followed by one of machine translation between two languages. We find appealing results across this range of tasks.

To appear at NIPS 2018; 9 pages with supplement

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Invertible Conditional GANs for image editing
• Variational Autoencoder for Deep Learning of Images, Labels and Captions
• Unsupervised Machine Translation Using Monolingual Corpora Only
• Towards Deeper Understanding of Variational Autoencoding Models
• Generalized Product of Experts for Automatic and Principled Fusion of Gaussian Process Predictions
• From Facial Parts Responses to Face Detection: A Deep Learning Approach
• Deep Variational Canonical Correlation Analysis
• Unsupervised Neural Machine Translation
• Generative Models of Visually Grounded Imagination

Cited by in corpus (14)

• Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations
• Hetero-Modal Variational Encoder-Decoder for Joint Modality Completion and Segmentation
• Modeling emotion in complex stories: the Stanford Emotional Narratives Dataset
• Predicting the Popularity of Micro-videos with Multimodal Variational Encoder-Decoder Framework
• Capturing Dynamics of Information Diffusion in SNS: A Survey of Methodology and Techniques
• A multimodal dynamical variational autoencoder for audiovisual speech representation learning
• Gaussian Process Prior Variational Autoencoders
• Imputation of missing values in multi-view data
• Normative Modeling using Multimodal Variational Autoencoders to Identify Abnormal Brain Structural Patterns in Alzheimer Disease
• Improving Normative Modeling for Multi-modal Neuroimaging Data using mixture-of-product-of-experts variational autoencoders
• Zero Shot Learning for Code Education: Rubric Sampling with Deep Learning Inference
• Composable Generative Models
• Latent Sensor Fusion: Multimedia Learning of Physiological Signals for Resource-Constrained Devices
• Recommending Burgers based on Pizza Preferences: Addressing Data Sparsity with a Product of Experts