Iterative Visual Reasoning Beyond Convolutions

Abstract

We present a novel framework for iterative visual reasoning. Our framework goes beyond current recognition systems that lack the capability to reason beyond stack of convolutions. The framework consists of two core modules: a local module that uses spatial memory to store previous beliefs with parallel updates; and a global graph-reasoning module. Our graph module has three components: a) a knowledge graph where we represent classes as nodes and build edges to encode different types of semantic relationships between them; b) a region graph of the current image where regions in the image are nodes and spatial relationships between these regions are edges; c) an assignment graph that assigns regions to classes. Both the local module and the global module roll-out iteratively and cross-feed predictions to each other to refine estimates. The final predictions are made by combining the best of both modules with an attention mechanism. We show strong performance over plain ConvNets, \eg achieving an $8.4\%$ absolute improvement on ADE measured by per-class average precision. Analysis also shows that the framework is resilient to missing regions for reasoning.

CVPR 2018

References in corpus (11)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Semi-Supervised Classification with Graph Convolutional Networks
• Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
• Deep Convolutional Networks on Graph-Structured Data
• A simple neural network module for relational reasoning
• Feature Pyramid Networks for Object Detection
• Beyond Skip Connections: Top-Down Modulation for Object Detection
• Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
• Visual Relationship Detection with Language Priors
• The More You Know: Using Knowledge Graphs for Image Classification
• Spatial Memory for Context Reasoning in Object Detection

Cited by in corpus (12)

• Relational Deep Reinforcement Learning
• Learning to Compose Dynamic Tree Structures for Visual Contexts
• Visual Semantic Reasoning for Image-Text Matching
• Efficient Coarse-to-Fine Non-Local Module for the Detection of Small Objects
• Learning Global and Local Consistent Representations for Unsupervised Image Retrieval via Deep Graph Diffusion Networks
• LinkNet: Relational Embedding for Scene Graph
• Graph-Aware Transformer: Is Attention All Graphs Need?
• Data-Efficient Graph Embedding Learning for PCB Component Detection
• HR-RCNN: Hierarchical Relational Reasoning for Object Detection
• Image-Level Attentional Context Modeling Using Nested-Graph Neural Networks
• Learning and Reasoning for Robot Sequential Decision Making under Uncertainty
• Universal-RCNN: Universal Object Detector via Transferable Graph R-CNN