Achieving Human Parity on Visual Question Answering

Abstract

The Visual Question Answering (VQA) task utilizes both visual image and language analysis to answer a textual question with respect to an image. It has been a popular research topic with an increasing number of real-world applications in the last decade. This paper describes our recent research of AliceMind-MMU (ALIbaba's Collection of Encoder-decoders from Machine IntelligeNce lab of Damo academy - MultiMedia Understanding) that obtains similar or even slightly better results than human being does on VQA. This is achieved by systematically improving the VQA pipeline including: (1) pre-training with comprehensive visual and textual feature representation; (2) effective cross-modal interaction with learning to attend; and (3) A novel knowledge mining framework with specialized expert modules for the complex VQA task. Treating different types of visual questions with corresponding expertise needed plays an important role in boosting the performance of our VQA architecture up to the human level. An extensive set of experiments and analysis are conducted to demonstrate the effectiveness of the new research work.

References in corpus (24)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Learning Transferable Visual Models From Natural Language Supervision
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• VL-BERT: Pre-training of Generic Visual-Linguistic Representations
• ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
• Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
• LXMERT: Learning Cross-Modality Encoder Representations from Transformers
• ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
• Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
• StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding
• Reading Wikipedia to Answer Open-Domain Questions
• InterBERT: Vision-and-Language Interaction for Multi-modal Pretraining
• UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning
• SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels
• MoVie: Revisiting Modulated Convolutions for Visual Counting and Beyond
• E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning
• Bilinear Graph Networks for Visual Question Answering
• Grid-VLP: Revisiting Grid Features for Vision-Language Pre-training
• ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integration