Modularized Textual Grounding for Counterfactual Resilience

Abstract

Computer Vision applications often require a textual grounding module with precision, interpretability, and resilience to counterfactual inputs/queries. To achieve high grounding precision, current textual grounding methods heavily rely on large-scale training data with manual annotations at the pixel level. Such annotations are expensive to obtain and thus severely narrow the model's scope of real-world applications. Moreover, most of these methods sacrifice interpretability, generalizability, and they neglect the importance of being resilient to counterfactual inputs. To address these issues, we propose a visual grounding system which is 1) end-to-end trainable in a weakly supervised fashion with only image-level annotations, and 2) counterfactually resilient owing to the modular design. Specifically, we decompose textual descriptions into three levels: entity, semantic attribute, color information, and perform compositional grounding progressively. We validate our model through a series of experiments and demonstrate its improvement over the state-of-the-art methods. In particular, our model's performance not only surpasses other weakly/un-supervised methods and even approaches the strongly supervised ones, but also is interpretable for decision making and performs much better in face of counterfactual classes than all the others.

arXiv comment13 pages, 12 figures, IEEE Conference on Computer Vision and Pattern Recognition, 2019

References in corpus (11)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Distributed Representations of Words and Phrases and their Compositionality
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• VQA: Visual Question Answering
• Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
• Weakly Supervised Object Localization with Multi-fold Multiple Instance Learning
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Learning to Reason: End-to-End Module Networks for Visual Question Answering
• MAttNet: Modular Attention Network for Referring Expression Comprehension
• Generating Counterfactual Explanations with Natural Language
• Weakly Supervised Attention Learning for Textual Phrases Grounding