Interpretable Visual Understanding with Cognitive Attention Network
arXiv:2108.02924
Abstract
While image understanding on recognition-level has achieved remarkable advancements, reliable visual scene understanding requires comprehensive image understanding on recognition-level but also cognition-level, which calls for exploiting the multi-source information as well as learning different levels of understanding and extensive commonsense knowledge. In this paper, we propose a novel Cognitive Attention Network (CAN) for visual commonsense reasoning to achieve interpretable visual understanding. Specifically, we first introduce an image-text fusion module to fuse information from images and text collectively. Second, a novel inference module is designed to encode commonsense among image, query and response. Extensive experiments on large-scale Visual Commonsense Reasoning (VCR) benchmark dataset demonstrate the effectiveness of our approach. The implementation is publicly available at https://github.com/tanjatang/CAN
ICANN21
References in corpus (6)
- DRAW: A Recurrent Neural Network For Image Generation
- ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
- Online and Customizable Fairness-aware Learning
- Using Machine Learning to Automate Mammogram Images Analysis
- Responsibility Management through Responsibility Networks
- Disentangled Dynamic Graph Deep Generation