Probabilistic Neural-symbolic Models for Interpretable Visual Question Answering

Abstract

We propose a new class of probabilistic neural-symbolic models, that have symbolic functional programs as a latent, stochastic variable. Instantiated in the context of visual question answering, our probabilistic formulation offers two key conceptual advantages over prior neural-symbolic models for VQA. Firstly, the programs generated by our model are more understandable while requiring lesser number of teaching examples. Secondly, we show that one can pose counterfactual scenarios to the model, to probe its beliefs on the programs that could lead to a specified answer given an image. Our results on the CLEVR and SHAPES datasets verify our hypotheses, showing that the model gets better program (and answer) prediction accuracy even in the low data regime, and allows one to probe the coherence and consistency of reasoning performed.

ICML 2019 Camera Ready + Appendix

Cited by in corpus (14)

• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• An Empirical Study on Leveraging Scene Graphs for Visual Question Answering
• MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
• Meta Module Network for Compositional Visual Reasoning
• Compositional Generalization via Neural-Symbolic Stack Machines
• Neuro-Symbolic Artificial Intelligence: Current Trends
• Bayesian Relational Memory for Semantic Visual Navigation
• NAAQA: A Neural Architecture for Acoustic Question Answering
• Iterated learning for emergent systematicity in VQA
• DMRM: A Dual-channel Multi-hop Reasoning Model for Visual Dialog
• Toward Interpretability of Dual-Encoder Models for Dialogue Response Suggestions
• Neuro-Symbolic VQA: A review from the perspective of AGI desiderata
• Domain-robust VQA with diverse datasets and methods but no target labels