Meta Module Network for Compositional Visual Reasoning

Abstract

Neural Module Network (NMN) exhibits strong interpretability and compositionality thanks to its handcrafted neural modules with explicit multi-hop reasoning capability. However, most NMNs suffer from two critical drawbacks: 1) scalability: customized module for specific function renders it impractical when scaling up to a larger set of functions in complex tasks; 2) generalizability: rigid pre-defined module inventory makes it difficult to generalize to unseen functions in new tasks/domains. To design a more powerful NMN architecture for practical use, we propose Meta Module Network (MMN) centered on a novel meta module, which can take in function recipes and morph into diverse instance modules dynamically. The instance modules are then woven into an execution graph for complex visual reasoning, inheriting the strong explainability and compositionality of NMN. With such a flexible instantiation mechanism, the parameters of instance modules are inherited from the central meta module, retaining the same model complexity as the function set grows, which promises better scalability. Meanwhile, as functions are encoded into the embedding space, unseen functions can be readily represented based on its structural similarity with previously observed ones, which ensures better generalizability. Experiments on GQA and CLEVR datasets validate the superiority of MMN over state-of-the-art NMN designs. Synthetic experiments on held-out unseen functions from GQA dataset also demonstrate the strong generalizability of MMN. Our code and model are released in Github https://github.com/wenhuchen/Meta-Module-Network.

arXiv commentAccepted to WACV 21 (Oral)

References in corpus (13)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• UNITER: UNiversal Image-TExt Representation Learning
• Hadamard Product for Low-rank Bilinear Pooling
• The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision
• Learning to Reason: End-to-End Module Networks for Visual Question Answering
• Scene Graph Generation by Iterative Message Passing
• Deep Modular Co-Attention Networks for Visual Question Answering
• Knowledge-Embedded Routing Network for Scene Graph Generation
• Probabilistic Neural-symbolic Models for Interpretable Visual Question Answering
• DVQA: Understanding Data Visualizations via Question Answering
• MUREL: Multimodal Relational Reasoning for Visual Question Answering

Cited by in corpus (3)

• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels
• VIOLIN: A Large-Scale Dataset for Video-and-Language Inference