Improved RAMEN: Towards Domain Generalization for Visual Question Answering

Abstract

Currently nearing human-level performance, Visual Question Answering (VQA) is an emerging area in artificial intelligence. Established as a multi-disciplinary field in machine learning, both computer vision and natural language processing communities are working together to achieve state-of-the-art (SOTA) performance. However, there is a gap between the SOTA results and real world applications. This is due to the lack of model generalisation. The RAMEN model \cite{Shrestha2019} aimed to achieve domain generalization by obtaining the highest score across two main types of VQA datasets. This study provides two major improvements to the early/late fusion module and aggregation module of the RAMEN architecture, with the objective of further strengthening domain generalization. Vector operations based fusion strategies are introduced for the fusion module and the transformer architecture is introduced for the aggregation module. Improvements of up to five VQA datasets from the experiments conducted are evident. Following the results, this study analyses the effects of both the improvements on the domain generalization problem. The code is available on GitHub though the following link \url{https://github.com/bhanukaManesha/ramen}.

arXiv comment11 pages, 3 figures, 2 tables

References in corpus (21)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• VQA: Visual Question Answering
• Dynamic Memory Networks for Visual and Textual Question Answering
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering
• Visual Question Answering: Datasets, Algorithms, and Future Challenges
• A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input
• LXMERT: Learning Cross-Modality Encoder Representations from Transformers
• A Focused Dynamic Attention Model for Visual Question Answering
• Compositional Attention Networks for Machine Reasoning
• Bilinear Attention Networks
• Training Recurrent Answering Units with Joint Loss Minimization for VQA
• C-VQA: A Compositional Split of the Visual Question Answering (VQA) v1.0 Dataset
• Dual Attention Networks for Multimodal Reasoning and Matching
• Attention on Attention: Architectures for Visual Question Answering (VQA)
• Learning Visual Knowledge Memory Networks for Visual Question Answering
• MMFT-BERT: Multimodal Fusion Transformer with BERT Encodings for Visual Question Answering
• Learning Visual Question Answering by Bootstrapping Hard Attention
• Learning Rich Image Region Representation for Visual Question Answering
• Self-Segregating and Coordinated-Segregating Transformer for Focused Deep Multi-Modular Network for Visual Question Answering
• Answer Them All! Toward Universal Visual Question Answering Models