DDRprog: A CLEVR Differentiable Dynamic Reasoning Programmer

Abstract

We present a novel Dynamic Differentiable Reasoning (DDR) framework for jointly learning branching programs and the functions composing them; this resolves a significant nondifferentiability inhibiting recent dynamic architectures. We apply our framework to two settings in two highly compact and data efficient architectures: DDRprog for CLEVR Visual Question Answering and DDRstack for reverse Polish notation expression evaluation. DDRprog uses a recurrent controller to jointly predict and execute modular neural programs that directly correspond to the underlying question logic; it explicitly forks subprocesses to handle logical branching. By effectively leveraging additional structural supervision, we achieve a large improvement over previous approaches in subtask consistency and a small improvement in overall accuracy. We further demonstrate the benefits of structural supervision in the RPN setting: the inclusion of a stack assumption in DDRstack allows our approach to generalize to long expressions where an LSTM fails the task.

References in corpus (6)

• Adam: A Method for Stochastic Optimization
• Deep Residual Learning for Image Recognition
• A simple neural network module for relational reasoning
• Stacked Attention Networks for Image Question Answering
• Learning to Reason: End-to-End Module Networks for Visual Question Answering
• CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning

Cited by in corpus (10)

• Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding
• Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning
• CLEVRER: CoLlision Events for Video REpresentation and Reasoning
• Systematic Generalization: What Is Required and Can It Be Learned?
• CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations
• Learning Visual Question Answering by Bootstrapping Hard Attention
• Giving Commands to a Self-driving Car: A Multimodal Reasoner for Visual Grounding
• Simulated Mental Imagery for Robotic Task Planning
• Hopper: Multi-hop Transformer for Spatiotemporal Reasoning
• Cascaded Mutual Modulation for Visual Reasoning