C3VQG: Category Consistent Cyclic Visual Question Generation

Abstract

Visual Question Generation (VQG) is the task of generating natural questions based on an image. Popular methods in the past have explored image-to-sequence architectures trained with maximum likelihood which have demonstrated meaningful generated questions given an image and its associated ground-truth answer. VQG becomes more challenging if the image contains rich contextual information describing its different semantic categories. In this paper, we try to exploit the different visual cues and concepts in an image to generate questions using a variational autoencoder (VAE) without ground-truth answers. Our approach solves two major shortcomings of existing VQG systems: (i) minimize the level of supervision and (ii) replace generic questions with category relevant generations. Most importantly, by eliminating expensive answer annotations, the required supervision is weakened. Using different categories enables us to exploit different concepts as the inference requires only the image and the category. Mutual information is maximized between the image, question, and answer category in the latent space of our VAE. A novel category consistent cyclic loss is proposed to enable the model to generate consistent predictions with respect to the answer category, reducing redundancies and irregularities. Additionally, we also impose supplementary constraints on the latent space of our generative model to provide structure based on categories and enhance generalization by encapsulating decorrelated features within each dimension. Through extensive experiments, the proposed model, C3VQG outperforms state-of-the-art VQG methods with weak supervision.

References in corpus (12)

• VQA: Visual Question Answering
• Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
• Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generation
• A Joint Model for Question Answering and Question Generation
• Learning Latent Subspaces in Variational Autoencoders
• Visual7W: Grounded Question Answering in Images
• Visual Question Generation as Dual Task of Visual Question Answering
• Neural Self Talk: Image Understanding via Continuous Questioning and Answering
• Automatic Generation of Grounded Visual Questions
• Learning Discriminative features using Center Loss and Reconstruction as Regularizer for Speech Emotion Recognition
• Product of Orthogonal Spheres Parameterization for Disentangled Representation Learning
• Hyperprior Induced Unsupervised Disentanglement of Latent Representations