Focal Visual-Text Attention for Visual Question Answering

Abstract

Recent insights on language and vision with neural networks have been successfully applied to simple single-image visual question answering. However, to tackle real-life question answering problems on multimedia collections such as personal photos, we have to look at whole collections with sequences of photos or videos. When answering questions from a large collection, a natural problem is to identify snippets to support the answer. In this paper, we describe a novel neural network called Focal Visual-Text Attention network (FVTA) for collective reasoning in visual question answering, where both visual and text sequence information such as images and text metadata are presented. FVTA introduces an end-to-end approach that makes use of a hierarchical process to dynamically determine what media and what time to focus on in the sequential data to answer the question. FVTA can not only answer the questions well but also provides the justifications which the system results are based upon to get the answers. FVTA achieves state-of-the-art performance on the MemexQA dataset and competitive results on the MovieQA dataset.

In CVPR 2018. Code, models and dataset are available here: https://memexqa.cs.cmu.edu/

References in corpus (13)

• ADADELTA: An Adaptive Learning Rate Method
• Teaching Machines to Read and Comprehend
• Bidirectional Attention Flow for Machine Comprehension
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• VQA: Visual Question Answering
• Dynamic Memory Networks for Visual and Textual Question Answering
• Machine Comprehension Using Match-LSTM and Answer Pointer
• Exploring Models and Data for Image Question Answering
• ReasoNet: Learning to Stop Reading in Machine Comprehension
• Learning a Text-Video Embedding from Incomplete and Heterogeneous Data
• A Compare-Aggregate Model for Matching Text Sequences
• Uncovering Temporal Context for Video Question and Answering
• MemexQA: Visual Memex Question Answering

Cited by in corpus (10)

• A Dual-Attention Learning Network with Word and Sentence Embedding for Medical Visual Question Answering
• Text as Neural Operator: Image Manipulation by Text Instruction
• Understanding the Role of Scene Graphs in Visual Question Answering
• SimAug: Learning Robust Representations from Simulation for Trajectory Prediction
• The Garden of Forking Paths: Towards Multi-Future Trajectory Prediction
• Technical Report of the Video Event Reconstruction and Analysis (VERA) System -- Shooter Localization, Models, Interface, and Beyond
• New Ideas and Trends in Deep Multimodal Content Understanding: A Review
• Iterative Context-Aware Graph Inference for Visual Dialog
• From Recognition to Prediction: Analysis of Human Action and Trajectory Prediction in Video
• Trying Bilinear Pooling in Video-QA