Cascaded Mutual Modulation for Visual Reasoning

Abstract

Visual reasoning is a special visual question answering problem that is multi-step and compositional by nature, and also requires intensive text-vision interactions. We propose CMM: Cascaded Mutual Modulation as a novel end-to-end visual reasoning model. CMM includes a multi-step comprehension process for both question and image. In each step, we use a Feature-wise Linear Modulation (FiLM) technique to enable textual/visual pipeline to mutually control each other. Experiments show that CMM significantly outperforms most related models, and reach state-of-the-arts on two visual reasoning benchmarks: CLEVR and NLVR, collected from both synthetic and natural languages. Ablation studies confirm that both our multistep framework and our visual-guided language modulation are critical to the task. Our code is available at https://github.com/FlamingHorizon/CMM-VR.

arXiv commentto appear in EMNLP 2018

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• VQA: Visual Question Answering
• A simple neural network module for relational reasoning
• Modulating early visual processing by language
• Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning
• Compositional Attention Networks for Machine Reasoning
• Learning to Compose Neural Networks for Question Answering
• Learning to Reason: End-to-End Module Networks for Visual Question Answering
• DDRprog: A CLEVR Differentiable Dynamic Reasoning Programmer
• Object Ordering with Bidirectional Matchings for Visual Reasoning