From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities
arXiv:2311.00308 · doi:10.1016/j.inffus.2024.102270
Abstract
The multimodal task of Visual Question Answering (VQA) encompassing elements of Computer Vision (CV) and Natural Language Processing (NLP), aims to generate answers to questions on any visual input. Over time, the scope of VQA has expanded from datasets focusing on an extensive collection of natural images to datasets featuring synthetic images, video, 3D environments, and various other visual inputs. The emergence of large pre-trained networks has shifted the early VQA approaches relying on feature extraction and fusion schemes to vision language pre-training (VLP) techniques. However, there is a lack of comprehensive surveys that encompass both traditional VQA architectures and contemporary VLP-based methods. Furthermore, the VLP challenges in the lens of VQA haven't been thoroughly explored, leaving room for potential open problems to emerge. Our work presents a survey in the domain of VQA that delves into the intricacies of VQA datasets and methods over the field's history, introduces a detailed taxonomy to categorize the facets of VQA, and highlights the recent trends, challenges, and scopes for improvement. We further generalize VQA to multimodal question answering, explore tasks related to VQA, and present a set of open problems for future investigation. The work aims to navigate both beginners and experts by shedding light on the potential avenues of research and expanding the boundaries of the field.
References in corpus (69)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Efficient Estimation of Word Representations in Vector Space
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- On the Opportunities and Risks of Foundation Models
- ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
- Microsoft COCO Captions: Data Collection and Evaluation Server
- YFCC100M: The New Data in Multimedia Research
- Flamingo: a Visual Language Model for Few-Shot Learning
- VisualBERT: A Simple and Performant Baseline for Vision and Language
- Hierarchical Question-Image Co-Attention for Visual Question Answering
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
- VL-BERT: Pre-training of Generic Visual-Linguistic Representations
- A Survey on Multimodal Large Language Models
- Multiple Object Recognition with Visual Attention
- Visual Instruction Tuning
- Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
- CoCa: Contrastive Captioners are Image-Text Foundation Models
- Exploring Models and Data for Image Question Answering
- SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
- Florence: A New Foundation Model for Computer Vision
- VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts
- Simple Baseline for Visual Question Answering
- Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
- Large-Scale Adversarial Training for Vision-and-Language Representation Learning
- ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering
- VLP: A Survey on Vision-Language Pre-training
- Visual Question Answering: Datasets, Algorithms, and Future Challenges
- Multimodal Residual Learning for Visual QA
- LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
- A Comprehensive Survey on Cross-modal Retrieval
- GIT: A Generative Image-to-text Transformer for Vision and Language
- Vision Transformer Adapter for Dense Predictions
- PaLI: A Jointly-Scaled Multilingual Language-Image Model
- Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models
- LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention
- Ground Truth Evaluation of Neural Network Explanations with CLEVR-XAI
- Language Is Not All You Need: Aligning Perception with Language Models
- Visual Entailment: A Novel Task for Fine-Grained Image Understanding
- Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks
- Show, Ask, Attend, and Answer: A Strong Baseline For Visual Question Answering
- Kosmos-2: Grounding Multimodal Large Language Models to the World
- FigureQA: An Annotated Figure Dataset for Visual Reasoning
- A Focused Dynamic Attention Model for Visual Question Answering
- LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model
- Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks
- Aligning where to see and what to tell: image caption with region-based attention and scene factorization
- Explicit Knowledge-based Reasoning for Visual Question Answering
- PathVQA: 30000+ Questions for Medical Visual Question Answering
- Visual Madlibs: Fill in the blank Image Generation and Question Answering
- Beyond Question-Based Biases: Assessing Multimodal Shortcut Learning in Visual Question Answering
- Towards Lightweight Transformer via Group-wise Transformation for Vision-and-Language Tasks
- R-VQA: Learning Visual Relation Facts with Semantic Attention for Visual Question Answering
- Zero-Shot Visual Question Answering
- PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering
- Change Detection Meets Visual Question Answering
- CLEVR-Dialog: A Diagnostic Dataset for Multi-Round Reasoning in Visual Dialog
- ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities
- Visual Question Answering for Cultural Heritage
- Gender and Racial Bias in Visual Question Answering Datasets
- Towards a Visual Turing Challenge
- On Evaluating Adversarial Robustness of Large Vision-Language Models
- Select, Substitute, Search: A New Benchmark for Knowledge-Augmented Visual Question Answering
- GRIT: General Robust Image Task Benchmark
- Language bias in Visual Question Answering: A Survey and Taxonomy
- HiVLP: Hierarchical Vision-Language Pre-Training for Fast Image-Text Retrieval
- A Picture May Be Worth a Hundred Words for Visual Question Answering
- Champion Solution for the WSDM2023 Toloka VQA Challenge
Cited by in corpus (3)
- FiCo-ITR: bridging fine-grained and coarse-grained image-text retrieval for comparative performance analysis
- Exploring the Application of Visual Question Answering (VQA) for Classroom Activity Monitoring
- KARAMEL: Knowledge-Aware Ranking for Multimodal Entity Linking in the Arts via Heterogeneous Graphs