BOK-VQA: Bilingual outside Knowledge-Based Visual Question Answering via Graph Representation Pretraining

Abstract

The current research direction in generative models, such as the recently developed GPT4, aims to find relevant knowledge information for multimodal and multilingual inputs to provide answers. Under these research circumstances, the demand for multilingual evaluation of visual question answering (VQA) tasks, a representative task of multimodal systems, has increased. Accordingly, we propose a bilingual outside-knowledge VQA (BOK-VQA) dataset in this study that can be extended to multilingualism. The proposed data include 17K images, 17K question-answer pairs for both Korean and English and 280K instances of knowledge information related to question-answer content. We also present a framework that can effectively inject knowledge information into a VQA system by pretraining the knowledge information of BOK-VQA data in the form of graph embeddings. Finally, through in-depth analysis, we demonstrated the actual effect of the knowledge information contained in the constructed training data on VQA.

References in corpus (9)

• NLTK: The Natural Language Toolkit
• Complex Embeddings for Simple Link Prediction
• A Novel Embedding Model for Knowledge Base Completion Based on Convolutional Neural Network
• Explicit Knowledge-based Reasoning for Visual Question Answering
• VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena
• TVQA: Localized, Compositional Video Question Answering
• ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering
• DramaQA: Character-Centered Video Story Understanding with Hierarchical QA
• CARETS: A Consistency And Robustness Evaluative Test Suite for VQA