Cross-Modal Generative Augmentation for Visual Question Answering

Abstract

Data augmentation has been shown to effectively improve the performance of multimodal machine learning models. This paper introduces a generative model for data augmentation by leveraging the correlations among multiple modalities. Different from conventional data augmentation approaches that apply low-level operations with deterministic heuristics, our method learns a generator that generates samples of the target modality conditioned on observed modalities in the variational auto-encoder framework. Additionally, the proposed model is able to quantify the confidence of augmented data by its generative probability, and can be jointly optimised with a downstream task. Experiments on Visual Question Answering as downstream task demonstrate the effectiveness of the proposed generative model, which is able to improve strong UpDn-based models to achieve state-of-the-art performance.

arXiv commentBMVC 2021

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Auto-Encoding Variational Bayes
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• VQA: Visual Question Answering
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• Neural Variational Inference and Learning in Belief Networks
• Pythia v0.1: the Winning Entry to the VQA Challenge 2018
• Self-Critical Reasoning for Robust Visual Question Answering
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Sequence-to-Sequence Data Augmentation for Dialogue Language Understanding