Variational Hetero-Encoder Randomized GANs for Joint Image-Text Modeling

Abstract

For bidirectional joint image-text modeling, we develop variational hetero-encoder (VHE) randomized generative adversarial network (GAN), a versatile deep generative model that integrates a probabilistic text decoder, probabilistic image encoder, and GAN into a coherent end-to-end multi-modality learning framework. VHE randomized GAN (VHE-GAN) encodes an image to decode its associated text, and feeds the variational posterior as the source of randomness into the GAN image generator. We plug three off-the-shelf modules, including a deep topic model, a ladder-structured image encoder, and StackGAN++, into VHE-GAN, which already achieves competitive performance. This further motivates the development of VHE-raster-scan-GAN that generates photo-realistic images in not only a multi-scale low-to-high-resolution manner, but also a hierarchical-semantic coarse-to-fine fashion. By capturing and relating hierarchical semantic and visual concepts with end-to-end training, VHE-raster-scan-GAN achieves state-of-the-art performance in a wide variety of image-text multi-modality learning and generation tasks.

arXiv commentICLR 2020

References in corpus (15)

• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Generative Adversarial Text to Image Synthesis
• VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning
• Beta-Negative Binomial Process and Poisson Factor Analysis
• IntroVAE: Introspective Variational Autoencoders for Photographic Image Synthesis
• Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks
• TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency
• Aligning where to see and what to tell: image caption with region-based attention and scene factorization
• Infinite Edge Partition Models for Overlapping Community Detection and Link Prediction
• PixelVAE: A Latent Variable Model for Natural Images
• Flow-GAN: Combining Maximum Likelihood and Adversarial Learning in Generative Models
• Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Language
• WHAI: Weibull Hybrid Autoencoding Inference for Deep Topic Modeling
• Recent Advances in Zero-shot Recognition
• Deep Latent Dirichlet Allocation with Topic-Layer-Adaptive Stochastic Gradient Riemannian MCMC