A First Look: Towards Explainable TextVQA Models via Visual and Textual Explanations

Abstract

Explainable deep learning models are advantageous in many situations. Prior work mostly provide unimodal explanations through post-hoc approaches not part of the original system design. Explanation mechanisms also ignore useful textual information present in images. In this paper, we propose MTXNet, an end-to-end trainable multimodal architecture to generate multimodal explanations, which focuses on the text in the image. We curate a novel dataset TextVQA-X, containing ground truth visual and multi-reference textual explanations that can be leveraged during both training and evaluation. We then quantitatively show that training with multimodal explanations complements model performance and surpasses unimodal baselines by up to 7% in CIDEr scores and 2% in IoU. More importantly, we demonstrate that the multimodal explanations are consistent with human interpretations, help justify the models' decision, and provide useful insights to help diagnose an incorrect prediction. Finally, we describe a real-world e-commerce application for using the generated multimodal explanations.

arXiv commentThis paper is done when Xingjian was an intern in Amazon PARS group, summer 2020. This paper is accepted by NAACL-MAI-Workshop, 2021

References in corpus (12)

• Semi-Supervised Classification with Graph Convolutional Networks
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations
• European Union regulations on algorithmic decision-making and a "right to explanation"
• VQA: Visual Question Answering
• Attention is not Explanation
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Explainable AI: Beware of Inmates Running the Asylum Or: How I Learnt to Stop Worrying and Love the Social and Behavioural Sciences
• Towards Transparent AI Systems: Interpreting Visual Question Answering Models
• Structured Multimodal Attentions for TextVQA
• Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and Captions
• Multi-Reference Training with Pseudo-References for Neural Translation and Text Generation

Cited by in corpus (1)

• Structured Multimodal Attentions for TextVQA