Generative Models of Visually Grounded Imagination

Abstract

It is easy for people to imagine what a man with pink hair looks like, even if they have never seen such a person before. We call the ability to create images of novel semantic concepts visually grounded imagination. In this paper, we show how we can modify variational auto-encoders to perform this task. Our method uses a novel training objective, and a novel product-of-experts inference network, which can handle partially specified (abstract) concepts in a principled and efficient way. We also propose a set of easy-to-compute evaluation metrics that capture our intuitive notions of what it means to have good visual imagination, namely correctness, coverage, and compositionality (the 3 C's). Finally, we perform a detailed comparison of our method with two existing joint image-attribute VAE methods (the JMVAE method of Suzuki et.al. and the BiVCCA method of Wang et.al.) by applying them to two datasets: the MNIST-with-attributes dataset (which we introduce here), and the CelebA dataset.

arXiv commentInternational Conference on Learning Representations (ICLR), 2018

References in corpus (6)

• Auto-Encoding Variational Bayes
• Variational Autoencoder for Deep Learning of Images, Labels and Captions
• Word Representations via Gaussian Embedding
• C-VQA: A Compositional Split of the Visual Question Answering (VQA) v1.0 Dataset
• Multimodal Word Distributions
• Autoencoders and Probabilistic Inference with Missing Data: An Exact Solution for The Factor Analysis Case

Cited by in corpus (25)

• Multimodal Generative Models for Scalable Weakly-Supervised Learning
• A survey of multimodal deep generative models
• The Conditional Entropy Bottleneck
• Small Sample Learning in Big Data Era
• Emergence of Compositional Language with Deep Generational Transmission
• Discrete and continuous representations and processing in deep learning: Looking forward
• Gaussian Process Prior Variational Autoencoders
• Scalable Gaussian Process Variational Autoencoders
• lamBERT: Language and Action Learning Using Multimodal BERT
• VAEs in the Presence of Missing Data
• Independent Subspace Analysis for Unsupervised Learning of Disentangled Representations
• Multimodal Generative Models for Compositional Representation Learning
• Disentangled VAE Representations for Multi-Aspect and Missing Data
• Zero Shot Learning for Code Education: Rubric Sampling with Deep Learning Inference
• CURI: A Benchmark for Productive Concept Learning Under Uncertainty
• A Simple Framework for Uncertainty in Contrastive Learning
• Conditional Inference in Pre-trained Variational Autoencoders via Cross-coding
• Dueling Decoders: Regularizing Variational Autoencoder Latent Spaces
• A Generative Map for Image-based Camera Localization
• Imitation Learning of Factored Multi-agent Reactive Models
• Multi-Source Neural Variational Inference
• Learning more expressive joint distributions in multimodal variational methods
• Variational learning across domains with triplet information
• Recommending Burgers based on Pizza Preferences: Addressing Data Sparsity with a Product of Experts
• Learn from Anywhere: Rethinking Generalized Zero-Shot Learning with Limited Supervision