Calibrating Concepts and Operations: Towards Symbolic Reasoning on Real Images
arXiv:2110.00519
Abstract
While neural symbolic methods demonstrate impressive performance in visual question answering on synthetic images, their performance suffers on real images. We identify that the long-tail distribution of visual concepts and unequal importance of reasoning steps in real data are the two key obstacles that limit the models' real-world potentials. To address these challenges, we propose a new paradigm, Calibrating Concepts and Operations (CCO), which enables neural symbolic models to capture underlying data characteristics and to reason with hierarchical importance. Specifically, we introduce an executor with learnable concept embedding magnitudes for handling distribution imbalance, and an operation calibrator for highlighting important operations and suppressing redundant ones. Our experiments show CCO substantially boosts the performance of neural symbolic methods on real images. By evaluating models on the real world dataset GQA, CCO helps the neural symbolic method NSCL outperforms its vanilla counterpart by 9.1% (from 47.0% to 56.1%); this result also largely reduces the performance gap between symbolic and non-symbolic methods. Additionally, we create a perturbed test set for better understanding and analyzing model performance on real images. Code is available at https://github.com/Lizw14/CaliCO.git .
arXiv commentTo appear in ICCV2021; Code at https://github.com/Lizw14/CaliCO.git
References in corpus (12)
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
- Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding
- Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning
- RUBi: Reducing Unimodal Biases in Visual Question Answering
- Compositional Attention Networks for Machine Reasoning
- Bilinear Attention Networks
- Attention Correctness in Neural Image Captioning
- Learning Perceptual Inference by Contrasting
- Neuro-Symbolic Visual Reasoning: Disentangling "Visual" from "Reasoning"
- Learning by Abstraction: The Neural State Machine
- Overcoming Language Priors in Visual Question Answering with Adversarial Regularization