Interpretable Visual Question Answering by Reasoning on Dependency Trees
arXiv:1809.01810
Abstract
Collaborative reasoning for understanding image-question pairs is a very critical but underexplored topic in interpretable visual question answering systems. Although very recent studies have attempted to use explicit compositional processes to assemble multiple subtasks embedded in questions, their models heavily rely on annotations or handcrafted rules to obtain valid reasoning processes, which leads to either heavy workloads or poor performance on compositional reasoning. In this paper, to better align image and language domains in diverse and unrestricted cases, we propose a novel neural network model that performs global reasoning on a dependency tree parsed from the question; thus, our model is called a parse-tree-guided reasoning network (PTGRN). This network consists of three collaborative modules: i) an attention module that exploits the local visual evidence of each word parsed from the question, ii) a gated residual composition module that composes the previously mined evidence, and iii) a parse-tree-guided propagation module that passes the mined evidence along the parse tree. Thus, PTGRN is capable of building an interpretable visual question answering (VQA) system that gradually derives image cues following question-driven parse-tree reasoning. Experiments on relational datasets demonstrate the superiority of PTGRN over current state-of-the-art VQA methods, and the visualization results highlight the explainable capability of our reasoning system.
arXiv comment14 pages, 10 figures. arXiv admin note: text overlap with arXiv:1804.00105
References in corpus (9)
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Hierarchical Question-Image Co-Attention for Visual Question Answering
- Dynamic Memory Networks for Visual and Textual Question Answering
- Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding
- Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning
- Learning to Count Objects in Natural Images for Visual Question Answering
- Hadamard Product for Low-rank Bilinear Pooling
- A Focused Dynamic Attention Model for Visual Question Answering
- Interpretable Counting for Visual Question Answering