Integrating Image Features with Convolutional Sequence-to-sequence Network for Multilingual Visual Question Answering

Abstract

Visual Question Answering (VQA) is a task that requires computers to give correct answers for the input questions based on the images. This task can be solved by humans with ease but is a challenge for computers. The VLSP2022-EVJVQA shared task carries the Visual Question Answering task in the multilingual domain on a newly released dataset: UIT-EVJVQA, in which the questions and answers are written in three different languages: English, Vietnamese and Japanese. We approached the challenge as a sequence-to-sequence learning task, in which we integrated hints from pre-trained state-of-the-art VQA models and image features with Convolutional Sequence-to-Sequence network to generate the desired answers. Our results obtained up to 0.3442 by F1 score on the public test set, 0.4210 on the private test set, and placed 3rd in the competition.

arXiv commentVLSP2022-EVJVQA

References in corpus (10)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Neural Machine Translation by Jointly Learning to Align and Translate
• Language Modeling with Gated Convolutional Networks
• ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
• OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework
• Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering
• KorQuAD1.0: Korean QA Dataset for Machine Reading Comprehension
• VLSP 2021 - ViMRC Challenge: Vietnamese Machine Reading Comprehension
• JaQuAD: Japanese Question Answering Dataset for Machine Reading Comprehension